On Thu, 17 Mar 2016, Robert LeBlanc wrote:
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA256
>
> I'm having trouble finding documentation about using ceph_test_rados. Can I
> run this on the existing cluster and will that provide useful info? It seems
> running it in the build will not have th
Hi all,
I encounter a trouble about cephfs sanpshot. It seems that the folder
'.snap' is exist.
But I use 'll -a' can't let it show up. And I enter that folder and create
folder in it, it showed something wrong to use snapshot.
Please check : http://imgur.com/elZhQvD
__
Hi,All.
I confirm the problem. When min_read_recency_for_promote> 1 data failure.
С уважением, Фасихов Ирек Нургаязович
Моб.: +79229045757
2016-03-17 15:26 GMT+03:00 Sage Weil :
> On Thu, 17 Mar 2016, Nick Fisk wrote:
> > There is got to be something else going on here. All that PR does is to
>
Also, is this ceph_test_rados rewriting objects quickly? I think that
the issue is with rewriting objects so if we can tailor the
ceph_test_rados to do that, it might be easier to reproduce.
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On Thu,
Neither of these file systems is recommended for production use underlying an
OSD. The general direction for ceph is to move away from having a file system
at all.
That effort is called "bluestore" and is supposed to show up in the jewel
release.
-H
> On Mar 18, 2016, at 11:15, Schlacta, Chr
The rule of thumb is to match the journal throughput to the OSD throughout.
I'm seeing ~180MB/s sequential write on my OSDs and I'm using one of the P3700
400GB units per six OSDs. The 400GB P3700 yields around 1200MB/s* and has
around 1/10th the latency of any SATA SSD I've tested.
I put a p
Hi,
Is there a tracker for this? We just hit the same problem on 10.0.5.
Cheers, Dan
# rpm -q ceph
ceph-10.0.5-0.el7.x86_64
# cat /etc/redhat-release
CentOS Linux release 7.2.1511 (Core)
# ceph-disk -v prepare /dev/sdc
DEBUG:ceph-disk:get_dm_uuid /dev/sdc uuid path is /sys/dev/block/8:32/dm/uu
Basically, the lookup process is:
try DIR_9/DIR_5/DIR_4/DIR_D/DIR_E/DIR_C/DIR_9/DIR_7...doesn't exist
try DIR_9/DIR_5/DIR_4/DIR_D/DIR_E/DIR_C/DIR_9/...doesn't exist
try DIR_9/DIR_5/DIR_4/DIR_D/DIR_E/DIR_C/...doesn't exist
try DIR_9/DIR_5/DIR_4/DIR_D/DIR_E/...does exist, object must be here
If DIR
On Wed, Mar 16, 2016 at 06:36:33AM +, Pavan Rallabhandi wrote:
> I find this to be discussed here before, but couldn¹t find any solution
> hence the mail. In RGW, for a bucket holding objects in the range of ~
> millions, one can find it to take for ever to delete the bucket(via
> radosgw-admin
Hi,
Because of a tiny mistake preventing deb packages to be built, v10.0.5 was
released shortly after v10.0.4 and is now the current development release. The
Stable release team[0] collectively decided to help by publishing development
packages[1], starting with v10.0.5.
The packages for v10.0
Hi,
I am facing issues with some of my rbd volumes since yesterday. Some of
them completely hang at some point before eventually resuming IO, may it be
a few minutes or several hours later.
First and foremost, my setup : I already detailed it on the mailing list
[0][1]. Some changes have been mad
> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
> Stephen Harker
> Sent: 16 March 2016 16:22
> To: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] SSDs for journals vs SSDs for a cache tier,
which is
> better?
>
> On 2016-02-17 11:07, C
> On 17 Mar 2016, at 17:28, Erik Schwalbe wrote:
>
> Hi,
>
> at the moment I do some tests with SSD's and ceph.
> My Question is, how to mount an SSD OSD? With or without discard option?
I recommend running without discard but running "fstrim" command every now and
then (depends on how fast
Hi Chen,
On Thu, Mar 17, 2016 at 12:40:28AM +, Chen, Xiaoxi wrote:
> It’s already there, in
> http://download.ceph.com/debian-hammer/pool/main/c/ceph/.
I can only see ceph*_0.94.6-1~bpo80+1_amd64.deb there. Debian wheezy would
be bpo70.
Cheers,
Chris
> On 3/17/16, 7:20 AM, "Chris Dunlop"
Hi,
It's true, partprobe works intermittently. I extracted the key
commands to show the problem:
[18:44]# /usr/sbin/sgdisk --new=2:0:20480M --change-name=2:'ceph
journal' --partition-guid=2:aa23e07d-e6b3-4261-a236-c0565971d88d
--typecode=2:45b0969e-9b03-4f30-b4c6-b4b80ceff106 --mbrtogpt --
/dev/s
Thanks all for your suggestions and advice. I'll let you know how it
goes :)
Stephen
On 2016-03-16 16:58, Heath Albritton wrote:
The rule of thumb is to match the journal throughput to the OSD
throughout. I'm seeing ~180MB/s sequential write on my OSDs and I'm
using one of the P3700 400GB uni
HI
i setup 2 clusters and in using radosgw_agent to sync them last week the
sync stop working if on runinig the agent from command line i see its stuck
on 2 files in the console im geting :
2016-03-17 21:11:57,391 14323 [radosgw_agent.worker][DEBUG ] op state is []
2016-03-17 21:11:57,391 14323 [ra
Hi John,
How to set this feature on?
Thank you
2016-03-17 21:41 GMT+08:00 Gregory Farnum :
> On Thu, Mar 17, 2016 at 3:49 AM, John Spray wrote:
> > Snapshots are disabled by default:
> >
> http://docs.ceph.com/docs/hammer/cephfs/early-adopters/#most-stable-configuration
>
> Which makes me wonde
I posted about this a while ago, and someone else has since inquired,
but I am seriously wanting to know if anybody has figured out how to
boot from a RBD device yet using ipxe or similar. Last I read.
loading the kernel and initrd from object storage would be
theoretically easy, and would only re
Hi,
I had no special logging activated.
Today I re-enabled exclusive-lock object-map and fast-diff on an image in 9.2.1
As soon as I ran an rbd export-diff I had lots of these error messages on the
console of the rbd export process:
2016-03-18 11:18:21.546658 7f77245d1700 1 heartbeat_map is_h
Great, I just recovered the first placement group from this error. To be
sure, I ran a deep-scrub and that comes back clean.
Thanks for all your help.
Regards,
Jeff
On Thu, Mar 17, 2016 at 11:58 AM, Samuel Just wrote:
> Oh, it's getting a stat mismatch. I think what happened is that on
> on
Snapshots are disabled by default:
http://docs.ceph.com/docs/hammer/cephfs/early-adopters/#most-stable-configuration
John
On Thu, Mar 17, 2016 at 10:02 AM, 施柏安 wrote:
> Hi all,
> I encounter a trouble about cephfs sanpshot. It seems that the folder
> '.snap' is exist.
> But I use 'll -a' can't l
Yep, thanks for all the help tracking down the root cause!
-Sam
On Thu, Mar 17, 2016 at 10:50 AM, Jeffrey McDonald wrote:
> Great, I just recovered the first placement group from this error. To be
> sure, I ran a deep-scrub and that comes back clean.
>
> Thanks for all your help.
> Regards,
>
Hi,
on ubuntu 14.04 client and centos 7.2 client with centos 7 Hammer
its working without problems.
--
Mit freundlichen Gruessen / Best regards
Oliver Dzombic
IP-Interactive
mailto:i...@ip-interactive.de
Anschrift:
IP Interactive UG ( haftungsbeschraenkt )
Zum Sonnenberg 1-3
63571 Gelnhause
Hi, Nick
I switched between forward and writeback. (forward -> writeback)
С уважением, Фасихов Ирек Нургаязович
Моб.: +79229045757
2016-03-17 16:10 GMT+03:00 Nick Fisk :
> > -Original Message-
> > From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
> > Irek Fasikhov
Hi Sam,
In the 70.459 logs from the deep-scrub, there is an error:
$ zgrep "= \-2$" ceph-osd.307.log.1.gz
2016-03-07 16:11:41.828332 7ff30cdad700 10
filestore(/var/lib/ceph/osd/ceph-307) remove
70.459s0_head/79ced459/default.724733.17__shadow_prostate/rnaseq/8e5da6e8-8881-4813-a4e3-327df57fd1b7/
On Wed, 16 Mar 2016, Eric Eastman wrote:
> Thank you for doing this. It will make testing 10.0.x easier for all of us
> in the field, and will make it easier to report bugs, as we will know that
> the problems we find were not caused by our build process.
Note that you can also always pull build
I'll miss the Ceph community as well. There was a few things I really
wanted to work in with Ceph.
I got this:
update_object_version oid 13 v 1166 (ObjNum 1028 snap 0 seq_num 1028)
dirty exists
1038: left oid 13 (ObjNum 1028 snap 0 seq_num 1028)
1040: finishing write tid 1 to nodez23350-256
10
Cherry-picking that commit onto v0.94.6 wasn't clean so I'm just
building your branch. I'm not sure what the difference between your
branch and 0.94.6 is, I don't see any commits against
osd/ReplicatedPG.cc in the last 5 months other than the one you did
today.
Robert LeBlanc
PGP F
Hi Nick,
Your solution requires manual configuration for each VM and cannot be setup as
part of an automated OpenStack deployment.
It would be really nice if it was a hypervisor based setting as opposed to a VM
based setting.
Thanks
Daniel
-Original Message-
From: Nick Fisk [mailto:
Thank you for doing this. It will make testing 10.0.x easier for all of us
in the field, and will make it easier to report bugs, as we will know that
the problems we find were not caused by our build process.
Eric
On Wed, Mar 16, 2016 at 7:14 AM, Loic Dachary wrote:
> Hi,
>
> Because of a tiny
On Thu, 17 Mar 2016, Nick Fisk wrote:
> There is got to be something else going on here. All that PR does is to
> potentially delay the promotion to hit_set_period*recency instead of
> just doing it on the 2nd read regardless, it's got to be uncovering
> another bug.
>
> Do you see the same pro
We are trying to figure out how to use rados bench to reproduce. Ceph
itself doesn't seem to think there is any corruption, but when you do a
verify inside the RBD, there is. Can rados bench verify the objects after
they are written? It also seems to be primarily the filesystem metadata
that is cor
> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
> Adrien Gillard
> Sent: 17 March 2016 10:23
> To: ceph-users
> Subject: [ceph-users] RBD hanging on some volumes of a pool
>
> Hi,
>
> I am facing issues with some of my rbd volumes since yes
Hi Nick,
Thank you for your feedback. The cache tiers was fine. We identified some
packet loss between two switches. As usual with network, relatively easy to
identify but not something that comes to mind at first :)
Adrien
On Thu, Mar 17, 2016 at 2:32 PM, Nick Fisk wrote:
> > -Original Me
For clusters that were created pre-hammer and want to use ceph-deploy to
create additional rgw instances is there a way to create the
bootstrap-rgw keyring?
http://docs.ceph.com/ceph-deploy/docs/rgw.html
--
Derek T. Yarnell
University of Maryland
Institute for Advanced Computer Studies
_
> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
> Irek Fasikhov
> Sent: 17 March 2016 13:00
> To: Sage Weil
> Cc: Robert LeBlanc ; ceph-users us...@lists.ceph.com>; Nick Fisk ; William Perkins
>
> Subject: Re: [ceph-users] data corruption w
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Possible, it looks like all the messages comes from a test suite. Is
there some logging that would expose this or an assert that could be
added? We are about ready to do some testing in our lab to see if we
can replicate it and workaround the issue.
just got done with a test against a build of 0.94.6 minus the two commits
that were backported in PR 7207. everything worked as it should with the
cache-mode set to writeback and the min_read_recency_for_promote set to 2.
assuming it works properly on master, there must be a commit that we're
missi
On 2016-02-17 11:07, Christian Balzer wrote:
On Wed, 17 Feb 2016 10:04:11 +0100 Piotr Wachowicz wrote:
> > Let's consider both cases:
> > Journals on SSDs - for writes, the write operation returns right
> > after data lands on the Journal's SSDs, but before it's written to
> > the backing HDD.
I have a cluster of around 630 OSDs with 3 dedicated monitors and 2
dedicated gateways. The entire cluster is running hammer (0.94.5
(9764da52395923e0b32908d83a9f7304401fee43)).
(Both of my gateways have stopped responding to curl right now.
root@host:~# timeout 5 curl localhost ; echo $?
124
Hi Stable Release Team for v0.94,
On Thu, Mar 10, 2016 at 11:00:06AM +1100, Chris Dunlop wrote:
> On Wed, Mar 02, 2016 at 06:32:18PM +0700, Loic Dachary wrote:
>> I think you misread what Sage wrote : "The intention was to
>> continue building stable releases (0.94.x) on the old list of
>> support
On Wednesday, 16 March 2016, Derek Yarnell wrote:
> Hi,
>
> We have a user with a 50GB quota and has now a single bucket with 20GB
> of files. They had previous buckets created and removed but the quota
> has not decreased. I understand that we do garbage collection but it
> has been significan
On 03/17/2016 03:51 AM, Schlacta, Christ wrote:
I posted about this a while ago, and someone else has since inquired,
but I am seriously wanting to know if anybody has figured out how to
boot from a RBD device yet using ipxe or similar. Last I read.
loading the kernel and initrd from object stor
On Mar 18, 2016 4:31 PM, "Lionel Bouton"
>
> Will bluestore provide the same protection against bitrot than BTRFS?
> Ie: with BTRFS the deep-scrubs detect inconsistencies *and* the OSD(s)
> with invalid data get IO errors when trying to read corrupted data and
> as such can't be used as the source
If you google "ceph bluestore" you'll be able to find a couple slide decks on
the topic. One of them by Sage is easy to follow without the benefit of the
presentation. There's also the " Redhat Ceph Storage Roadmap 2016" deck.
In any case, bluestore is not intended to address bitrot. Given th
Try the following:
# rbd lock remove vm-114-disk-1 "auto 140454012457856" client.71260575
--
Jason Dillaman
- Original Message -
> From: "Christoph Adomeit"
> To: ceph-us...@ceph.com
> Sent: Friday, March 18, 2016 11:14:00 AM
> Subject: [ceph-users] Cannot remove rbd locks
>
> Hi,
Hello,
I have a problem with the following crushmap :
# begin crush map
tunable choose_local_tries 0
tunable choose_local_fallback_tries 0
tunable choose_total_tries 50
tunable chooseleaf_descend_once 1
tunable straw_calc_version 1
# devices
device 0 osd.0
device 1 osd.1
device 2 osd.2
device 3
robert and i have done some further investigation the past couple days on
this. we have a test environment with a hard drive tier and an ssd tier as
a cache. several vms were created with volumes from the ceph cluster. i did
a test in each guest where i un-tarred the linux kernel source multiple
ti
Hi,
some of my rbds show they have an exclusive lock.
I think the lock can be stale or weeks old.
We have also once added feature exclusive lock and later removed that feature
I can see the lock:
root@machine:~# rbd lock list vm-114-disk-1
There is 1 exclusive lock on this image.
Locker
What OS are you using?
I have a lot more open connections than that. (though i have some other
issues, where rgw sometimes returns 500 errors, it doesn't stop like yours)
You might try tuning civetweb's num_threads and 'rgw num rados handles':
rgw frontends = civetweb num_threads=125
error_log_f
Hi Sam,
I've written a script but i'm a little leary of unleasing it until I find a
few more cases to test. The script successfully removed the file
mentioned above.
I took the next pg which was marked inconsistent and ran the following
command over those pg directory structures:
find . -name "
OK, I think I have it now. I do have one more question, in this case, the
hash indicates the directory structure but how do I know from the hash how
many levels I should go down.If the hash is a 32-bit hex integer, *how
do I know how many should be included as part of the hash for the directo
Hi,
at the moment I do some tests with SSD's and ceph.
My Question is, how to mount an SSD OSD? With or without discard option?
Where should I do the fstrim, when I mount the OSD without discard? On the ceph
storage node? Inside the vm, running on rbd?
What is the best practice there.
Than
List,
We have stood up a Infernalis 9.2.0 cluster on RHEL 7.2. We are using the
radosGW to store potentially billions of small to medium sized objects (64k
- 1MB).
We have run into an issue twice thus far where .rgw.bucket.index placement
groups will become corrupt during recovery after a drive
On 3/17/16 1:41 PM, Marius Vaitiekunas wrote:
> It's possible that somebody changed the owner of some bucket. But all
> objects in that bucket still belongs to this user. That way you can get
> quota exceeded. We had the same situation.
Well the user says he didn't write to any other buckets than
Hi,
we have upgraded our ceph-cluster to infernalis from hammer.
Ceph is still running as root and we are using the
"setuser match path = /var/lib/ceph/$type/$cluster-$id" directive in ceph.conf
Now we would like to change the ownership of data-files and devices to ceph at
runtime.
What ist t
FWIW, from purely a performance perspective Ceph usually looks pretty
fantastic on a fresh BTRFS filesystem. In fact it will probably
continue to look great until you do small random writes to large objects
(like say to blocks in an RBD volume). Then COW starts fragmenting the
objects into ob
Thanks Dan, I have raised the tracker for this issue
http://tracker.ceph.com/issues/15176
On Thu, Mar 17, 2016 at 10:47 AM, Dan van der Ster
wrote:
> Hi,
>
> It's true, partprobe works intermittently. I extracted the key
> commands to show the problem:
>
> [18:44]# /usr/sbin/sgdisk --new=2:0:204
> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
> Daniel Niasoff
> Sent: 16 March 2016 21:02
> To: Nick Fisk ; 'Van Leeuwen, Robert'
> ; 'Jason Dillaman'
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] Local SSD cache for ceph o
On Tue, Mar 15, 2016 at 11:36 PM, Pavan Rallabhandi
wrote:
> Hi,
>
> I find this to be discussed here before, but couldn¹t find any solution
> hence the mail. In RGW, for a bucket holding objects in the range of ~
> millions, one can find it to take for ever to delete the bucket(via
> radosgw-admi
CfP 11th Workshop on Virtualization in High-Performance Cloud
Computing (VHPC '16)
CALL FOR PAPERS
11th Workshop on Virtualization in High-Performance Cloud Computing (VHPC '16)
held in conjunction with the International Supe
On 20/03/2016 3:38 AM, Heath Albritton wrote:
Ceph protects against bitrot at a much higher level by validating the
checksum of the entire placement group during a deep scrub.
Ceph has checksums? I didn't think it did.
Its my understanding that it just compares blocks between replications
Le 19/03/2016 18:38, Heath Albritton a écrit :
> If you google "ceph bluestore" you'll be able to find a couple slide
> decks on the topic. One of them by Sage is easy to follow without the
> benefit of the presentation. There's also the " Redhat Ceph Storage
> Roadmap 2016" deck.
>
> In any case
Hi John,
Really thank you for your help, and sorry about that I ask such a stupid
question of setting...
So isn't this feature ready in Jewel? I found something info says that the
features(snapshot, quota...) become stable in Jewel
Thank you
2016-03-18 21:07 GMT+09:00 John Spray :
> On Fri, Mar
On Fri, Mar 18, 2016 at 1:33 AM, 施柏安 wrote:
> Hi John,
> How to set this feature on?
ceph mds set allow_new_snaps true --yes-i-really-mean-it
John
> Thank you
>
> 2016-03-17 21:41 GMT+08:00 Gregory Farnum :
>>
>> On Thu, Mar 17, 2016 at 3:49 AM, John Spray wrote:
>> > Snapshots are disabled by
On Thu, Mar 17, 2016 at 3:49 AM, John Spray wrote:
> Snapshots are disabled by default:
> http://docs.ceph.com/docs/hammer/cephfs/early-adopters/#most-stable-configuration
Which makes me wonder if we ought to be hiding the .snaps directory
entirely in that case. I haven't previously thought about
Yes, I`m missing protection from Ceph too.
http://lists.ceph.com/pipermail/ceph-users-ceph.com/2016-February/007680.html
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
There is a directory structure hash, it's just that it's at the end of
the name and you'll have to check the xattr I mentioned to find it.
I think that file is actually the one we are talking about removing.
./DIR_9/DIR_5/DIR_4/DIR_D/default.724733.17\u\ushadow\uprostate\srnaseq\s8e5da6e8-8881-48
Is there documentation on all the steps showing how to upgrade from .94 to
10.0.5?
Thanks
Rick
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
There is got to be something else going on here. All that PR does is to
potentially delay the promotion to hit_set_period*recency instead of just doing
it on the 2nd read regardless, it's got to be uncovering another bug.
Do you see the same problem if the cache is in writeback mode before you s
Oh, it's getting a stat mismatch. I think what happened is that on
one of the earlier repairs it reset the stats to the wrong value (the
orphan was causing the primary to scan two objects twice, which
matches the stat mismatch I see here). A pg repair repair will clear
that up.
-Sam
On Thu, Mar
Yep, let me pull and build that branch. I tried installing the dbg
packages and running it in gdb, but it didn't load the symbols.
Robert LeBlanc
PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1
On Thu, Mar 17, 2016 at 11:36 AM, Sage Weil wrote:
> On Thu, 17 Ma
Hello,
On Sun, 20 Mar 2016 00:45:47 +0100 Lionel Bouton wrote:
> Le 19/03/2016 18:38, Heath Albritton a écrit :
> > If you google "ceph bluestore" you'll be able to find a couple slide
> > decks on the topic. One of them by Sage is easy to follow without the
> > benefit of the presentation. Th
On Thu, 17 Mar 2016, Robert LeBlanc wrote:
> We are trying to figure out how to use rados bench to reproduce. Ceph
> itself doesn't seem to think there is any corruption, but when you do a
> verify inside the RBD, there is. Can rados bench verify the objects after
> they are written? It also seems
Ok, like I said, most files with _long at the end are *not orphaned*.
The generation number also is *not* an indication of whether the file
is orphaned -- some of the orphaned files will have
as the generation number and others won't. For each long filename
object in a pg you woul
We would be a big user of this. We delete large buckets often and it takes
forever.
Though didn't I read that 'object expiration' support is on the near-term
RGW roadmap? That may do what we want.. we're creating thousands of objects
a day, and thousands of objects a day will be expiring, so RGW w
77 matches
Mail list logo
