Re: [ceph-users] OSDs cpu usage

2016-12-12 Thread David Riedl
10.2.5 exists because of this bug. Here are the patch notes http://docs.ceph.com/docs/master/release-notes/#v10-2-5-jewel Regards David Am 12.12.2016 um 17:09 schrieb George Kissandrakis: I saw that 10.2.5 is out but if a bug appeared on 10.2.4 would that have been fixed in 10.2.5, or

Re: [ceph-users] cephfs quota

2016-12-15 Thread David Disseldorp
nt, and is currently only supported by ceph-fuse. Cheers, David ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] cephfs quota

2016-12-16 Thread David Disseldorp
thin the filesystem? Also, keep in mind that enforcement doesn't occur immediately. Normally a short amount of time needs to elapse before the EDQUOT error is returned to the writer that exceeded the quota limit. Cheers, David ___ ceph-users mailing l

Re: [ceph-users] cephfs quota

2016-12-16 Thread David Disseldorp
Hi Matthew, On Fri, 16 Dec 2016 12:30:06 +, Matthew Vernon wrote: > Hello, > On 15/12/16 10:25, David Disseldorp wrote: > > > Are you using the Linux kernel CephFS client (mount.ceph), or the > > userspace ceph-fuse back end? Quota enforcement is performed by t

Re: [ceph-users] fio librbd result is poor

2016-12-19 Thread David Turner
EXT4 (as all my production clusters). >>> >>> The 2 SSD ODS nodes have 1x 200GB DC S3610 (OS and 4 journal partitions) >>> and 2x 400GB DC S3610s (2 180GB partitions, so 8 SSD OSDs total), same >>> specs as the HDD nodes otherwise. >>> Also one node with XFS, the

Re: [ceph-users] ceph program uses lots of memory

2016-12-29 Thread David Turner
least used osd is within 2% in those clusters. We have 99.9% of our data in 1 pool. [cid:imageef823d.JPG@9dc05da4.41ac2656]<https://storagecraft.com> David Turner | Cloud Operations Engineer | StorageCraft Technology Corporation<https://storage

Re: [ceph-users] ceph program uses lots of memory

2016-12-29 Thread David Turner
I accidentally replied to the wrong thread. this was meant for a different one. [cid:image5d7d18.JPG@87f9fba8.42bf12bd]<https://storagecraft.com> David Turner | Cloud Operations Engineer | StorageCraft Technology Corporation<https://storagecraft

Re: [ceph-users] linux kernel version for clients

2016-12-31 Thread David Turner
have received this message in error please notify us >> at once by return email and then delete both messages. We accept no >> liability for the distribution of viruses or similar in electronic >> communications. This notice should not be removed. >>

Re: [ceph-users] performance with/without dmcrypt OSD

2017-01-02 Thread David Turner
ss or you don't. If you don't then why use it? [cid:imagead2398.JPG@a71bea74.4b9b522b]<https://storagecraft.com> David Turner | Cloud Operations Engineer | StorageCraft Technology Corporation<https://storagecraft.com> 380 Data Drive Sui

Re: [ceph-users] cephfs ata1.00: status: { DRDY }

2017-01-05 Thread David Welch
rs running Kernel 4.5.5 Maybe it will cause the VM to refuse IO and has to be restarted. Maybe not and it will continue. Any input is appriciated. Thank you ! -- ~~ David Welch DevOps ARS http://thinkars.com ___ ceph-users mailing list ceph-us

Re: [ceph-users] new user error

2017-01-06 Thread David Turner
It should happen automatically when you start the node. Please provide ceph status, ceph osd tree, and any other pertinent specifics you can think of. [cid:image2cfdf2.JPG@41df26b2.449ae9c6]<https://storagecraft.com> David Turner | Cloud Oper

[ceph-users] cephfs AND rbds

2017-01-06 Thread David Turner
Can cephfs and rbds use the same pool to store data? I know you would need a separate metadata pool for cephfs, but could they share the same data pool? [cid:image90f904.JPG@bd76911d.4db65e9b]<https://storagecraft.com> David Turner | Cloud Oper

Re: [ceph-users] cephfs AND rbds

2017-01-07 Thread David Turner
. My osds aren't to a point of having too many PGs on them, I just wanted to mitigate the memory need of the osd processes. [cid:image503057.JPG@dc3fe83b.43b7c2e9]<https://storagecraft.com> David Turner | Cloud Operations Engineer | Storage

Re: [ceph-users] SSDs for data drives

2018-07-11 Thread David Blundell
drives which currently seem best to avoid… David > On 11 Jul 2018, at 11:57, Paul Emmerich wrote: > > Hi, > > we‘ve no long-term data for the SM variant. > Performance is fine as far as we can tell, but the main difference between > these two models should be endurance. &

Re: [ceph-users] SSDs for data drives

2018-07-11 Thread leo David
;> >> ___ >> ceph-users mailing list >> ceph-users@lists.ceph.com >> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com >> > > > ___ > ce

Re: [ceph-users] PGs stuck peering (looping?) after upgrade to Luminous.

2018-07-12 Thread David Majchrzak
think we ran into this once) if your mons and osds are different versions. // david On jul 12 2018, at 11:45 am, Magnus Grönlund wrote: > > Hi list, > > Things went from bad to worse, tried to upgrade some OSDs to Luminous to see > if that could help but that didn’t appear to make

[ceph-users] OSD fails to start after power failure

2018-07-14 Thread David Young
Hey folks, I have a Luminous 12.2.6 cluster which suffered a power failure recently. On recovery, one of my OSDs is continually crashing and restarting, with the error below: 9ae00 con 0     -3> 2018-07-15 09:50:58.313242 7f131c5a9700 10 monclient: tick     -2> 2018-07-15 09:50:58.313277

[ceph-users] OSD fails to start after power failure (with FAILED assert(num_unsent <= log_queue.size()) error)

2018-07-14 Thread David Young
Hey folks, Sorry, posting this from a second account, since for some reason my primary account doesn't seem tobeable to post to the list... I have a Luminous 12.2.6 cluster which suffered a power failure recently. On recovery, one of my OSDs is continually crashing and restarting, with the e

Re: [ceph-users] Cephfs meta data pool to ssd and measuring performance difference

2018-07-30 Thread David C
Something like smallfile perhaps? https://github.com/bengland2/smallfile Or you just time creating/reading lots of files With read benching you would want to ensure you've cleared your mds cache or use a dataset larger than the cache. I'd be interested in seeing your results, I this on the to do

Re: [ceph-users] CephFS Snapshots in Mimic

2018-07-31 Thread David Disseldorp
# cat .snap/mysnapshot/data before snap rapido1:/mnt/cephfs# cat data before snap after snap Cheers, David ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Luminous OSD crashes every few seconds: FAILED assert(0 == "past_interval end mismatch")

2018-08-01 Thread J David
Hello all, On Luminous 12.2.7, during the course of recovering from a failed OSD, one of the other OSDs started repeatedly crashing every few seconds with an assertion failure: 2018-08-01 12:17:20.584350 7fb50eded700 -1 log_channel(cluster) log [ERR] : 2.621 past_interal bound [19300,21449) end d

[ceph-users] Error: journal specified but not allowed by osd backend

2018-08-01 Thread David Majchrzak
d set setuser_match_path in there instead for the old OSDs? Should I do any other steps preceding this if I want to use the same osd UUID? I've only stopped ceph-osd@21, removed the physical disk, inserted new one and tried running prepare. Kind Regards, David ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Luminous OSD crashes every few seconds: FAILED assert(0 == "past_interval end mismatch")

2018-08-01 Thread J David
seemed very happy to see it again. Not sure if this solution works generally or if it was specific to this case, or if it was not a solution and the cluster will eat itself overnight. But, so far so good! Thanks! On Wed, Aug 1, 2018 at 3:42 PM, J David wrote: > Hello all, > > On

Re: [ceph-users] Luminous OSD crashes every few seconds: FAILED assert(0 == "past_interval end mismatch")

2018-08-01 Thread J David
On Wed, Aug 1, 2018 at 9:53 PM, Brad Hubbard wrote: > What is the status of the cluster with this osd down and out? Briefly, miserable. All client IO was blocked. 36 pgs were stuck “down.” pg query reported that they were blocked by that OSD, despite that OSD not holding any replicas for them,

Re: [ceph-users] Error: journal specified but not allowed by osd backend

2018-08-02 Thread David Majchrzak
Hm. You are right. Seems ceph-osd uses id 0 in main.py. I'll have a look in my dev cluster and see if it helps things. /usr/lib/python2.7/dist-packages/ceph_disk/main.py def check_journal_reqs(args): _, _, allows_journal = command([ 'ceph-osd', '--check-allows-journal', '-i', '0', '--log-file', '$

[ceph-users] RGW problems after upgrade to Luminous

2018-08-02 Thread David Turner
I upgraded my last cluster to Luminous last night. It had some very large bucket indexes on Jewel which caused a couple problems with the upgrade, but finally everything finished and we made it to the other side, but now I'm having problems with [1] these errors populating a lot of our RGW logs an

Re: [ceph-users] Error: journal specified but not allowed by osd backend

2018-08-03 Thread David Majchrzak
that I didn't have to backfill twice then, by reusing the osd uuid. I'll see if I can add to the docs after we have updated to Luminous or Mimic and started using ceph-volume. Kind Regards David Majchrzak On aug 3 2018, at 4:16 pm, Eugen Block wrote: > > Hi, > we have a ful

Re: [ceph-users] RGW problems after upgrade to Luminous

2018-08-03 Thread David Turner
I am currently unable to write any data to this bucket in this current state. Does anyone have any ideas for reverting to the original index shards and cancel the reshard processes happening to the bucket? On Thu, Aug 2, 2018 at 12:32 PM David Turner wrote: > I upgraded my last cluster

Re: [ceph-users] RGW problems after upgrade to Luminous

2018-08-03 Thread David Turner
; bucket index. > > Let me know if it happens to fix the issue for you. > > Yehuda. > > On Fri, Aug 3, 2018 at 9:46 AM, Yehuda Sadeh-Weinraub > wrote: > > Is it actually resharding, or is it just stuck in that state? > > > > On Fri, Aug 3, 2018 at 7:55 AM, David

Re: [ceph-users] Testing a hypothetical crush map

2018-08-06 Thread David Majchrzak
.com%2Fblog%2F2015%2F02%2F02%2Fcrushmap-example-of-a-hierarchical-cluster-map&recipient=Y2VwaC11c2Vyc0BsaXN0cy5jZXBoLmNvbQ%3D%3D) David Majchrzak CTO ODERLAND Webbhotell AB E // da...@oderland.se (https://link.getmailspring.com/link/1533557996.local-a293f1fe-4d41-v1.3.0-fd741...@getmailspring.com/2?

Re: [ceph-users] rhel/centos7 spectre meltdown experience

2018-08-16 Thread David Turner
The general consensus when this came out was that Ceph clusters shouldn't be visible enough in your infrastructure to worry about vulnerabilities from external sources. I went ahead and upgraded to patch some of my clusters and didn't see any performance problems with them. Benchmarks showed an i

Re: [ceph-users] BlueStore upgrade steps broken

2018-08-17 Thread David Turner
> ceph-volume lvm create --osd-id 0 --bluestore --data /dev/sdc --block.db /dev/sdb --block.wal /dev/sdb That command can't work... You're telling it to use the entire /dev/sdb device for the db and then again to do it for the wal, but you can only use the entire device once. There are 2 things w

Re: [ceph-users] Upgrade to Infernalis: failed to pick suitable auth object

2018-08-17 Thread David Turner
In your baby step upgrade you should avoid the 2 non-LTS releases of Infernalis and Kraken. You should go from Hammer to Jewel to Luminous. The general rule of doing the upgrade to put all of your OSDs to be owned by ceph was to not change the ownership as part of the upgrade. There is a [1] con

Re: [ceph-users] RBD journal feature

2018-08-17 Thread David Turner
> I'll open a tracker ticket to fix the issue. Is there a tracker URL we can follow along with? On Thu, Aug 16, 2018 at 10:04 PM Glen Baars wrote: > Thanks for your help 😊 > > Kind regards, > > *Glen Baars* > > *From:* Jason Dillaman > *Sent:* Thursday, 16 August 2018 10:21 PM > > > *To:* Glen

Re: [ceph-users] BlueStore upgrade steps broken

2018-08-17 Thread David Turner
Searching has yielded nothing. >> >> We are working on expanding the tooling to this for you, but until >> then, it is up to the user to create the LVs manually. >> >> This section might help out a bit on what you would need (for block.db): >> >>

Re: [ceph-users] BlueStore upgrade steps broken

2018-08-17 Thread David Turner
Does the block and/or wal partition need to be an LV? I just passed ceph-volume the raw partition and it seems to be working fine. On Fri, Aug 17, 2018 at 2:54 PM Alfredo Deza wrote: > On Fri, Aug 17, 2018 at 10:24 AM, Robert Stanford > wrote: > > > > I was using the ceph-volume create comman

Re: [ceph-users] BlueStore upgrade steps broken

2018-08-17 Thread David Turner
Correct, I meant a partition of a block device. By 'raw partition I just meant that there wasn't any LVM layer like in your example. Thanks Alfredo. On Fri, Aug 17, 2018 at 3:05 PM Alfredo Deza wrote: > On Fri, Aug 17, 2018 at 2:55 PM, David Turner > wrote: > > Doe

Re: [ceph-users] how can time machine know difference between cephfs fuse and kernel client?

2018-08-17 Thread David Turner
It's probably not that time machine or windows can tell which is which, but there are some features that the kernel driver does not support like quotas. Perhaps the way quotas are managed and reported by the fuse client are responding with the 0 bytes free. I'm not sure exactly. I use the fuse c

Re: [ceph-users] Upgrade to Infernalis: failed to pick suitable auth object

2018-08-18 Thread David Turner
The reason to separate the items is to make one change at a time so you know what might have caused your problems. Good luck. On Sat, Aug 18, 2018, 4:52 AM Kees Meijs wrote: > Hi David, > > Thank you for pointing out the option. > > On http://docs.ceph.com/docs/infernalis/release

Re: [ceph-users] Upgrade to Infernalis: failed to pick suitable auth object

2018-08-18 Thread David Turner
You can't change the file permissions while the pads are still running. The instructions you pasted earlier said to stop the osd and run the chmod. What do your osd logs of the primary osd say about the PGs that are inconsistent? On Sat, Aug 18, 2018, 11:43 AM Kees Meijs wrote: > Hi again, > > A

Re: [ceph-users] MDS stuck in 'rejoin' after network fragmentation caused OSD flapping

2018-08-19 Thread David Turner
I second that you do not have nearly enough RAM in these servers and I don't you have at least 72 CPU cores either which means you again don't have the minimum recommendation for the amount of OSDs you have, let alone everything else. I would suggest you start by moving your MDS daemons off of the

Re: [ceph-users] BlueStore upgrade steps broken

2018-08-19 Thread David Turner
On 18 August 2018 at 03:06, David Turner wrote: > >> The WAL will choose the fastest device available. >> > > Any idea how it makes this determination automatically? is it doing a > hdparm -t or similar? is fastest=bandwidth, IOPs or latency? > > > > ___

Re: [ceph-users] Librados Keyring Issues

2018-08-19 Thread David Turner
You are not specifying which user you are using. Your config file specifies the keyring, but it's still trying to use the default user admin. If you specify that in your python you'll be good to go. On Sun, Aug 19, 2018, 9:17 PM Benjamin Cherian wrote: > Hi, > > I'm trying to write a simple test

Re: [ceph-users] Librados Keyring Issues

2018-08-20 Thread David Turner
on when using keyring containing key for dms > user! > > Regards, > > Benjamin Cherian > > On Sun, Aug 19, 2018 at 9:55 PM, Benjamin Cherian < > benjamin.cher...@gmail.com> wrote: > >> Hi David, >> >> Thanks for the reply...I had thought there mig

Re: [ceph-users] Tons of "cls_rgw.cc:3284: gc_iterate_entries end_key=" records in OSD logs

2018-08-20 Thread David Turner
I'm assuming you use RGW and that you have a GC pool for RGW. It also might beat assumed that your GC pool only has 8 PGs. Are any of those guesses correct? On Mon, Aug 20, 2018, 5:13 AM Jakub Jaszewski wrote: > Issue tracker http://tracker.ceph.com/issues/23801. > Still don't know why only par

Re: [ceph-users] Upgrade to Infernalis: failed to pick suitable auth object

2018-08-20 Thread David Turner
My suggestion would be to remove the osds and let the cluster recover from all of the other copies. I would deploy the node back to Hammer instead of Infernalis. Either that or remove these osds, let the cluster backfill, and then upgrade to Jewel, and then luminous, and maybe mimic if you're plann

Re: [ceph-users] Set existing pools to use hdd device class only

2018-08-20 Thread David Turner
All of the data moves because all of the crush IDs for the hosts and osds changes when you configure a crush rule to only use SSDs or HDDs. Crush creates shadow hosts and shadow osds in the crush map that only have each class of osd. So if you had node1 with osd.0 as an hdd and osd.1 as an SSD, th

Re: [ceph-users] Tons of "cls_rgw.cc:3284: gc_iterate_entries end_key=" records in OSD logs

2018-08-20 Thread David Turner
x27;t know why there logs are being spammed with GC messages though. Hopefully someone else can shed a light on that. I cc'd Yehuda on this, the primary RGW dev. On Mon, Aug 20, 2018, 7:09 AM Jakub Jaszewski wrote: > Hi David, > > Right, we use this cluster (v12.2.5, fresh installatio

[ceph-users] Removing all rados objects based on a prefix

2018-08-20 Thread David Turner
The general talk about the rados cleanup command is to clean things up after benchmarking. Could this command also be used for deleting an old RGW bucket or an RBD. For instance, a bucket with a prefix of `25ff9eff-058b-41e3-8724-cfffecb979c0.9709451.1` such that all objects in the default.rgw.bu

Re: [ceph-users] ceph balancer: further optimizations?

2018-08-20 Thread David Turner
You might just have too much data per PG. If a single PG can account for 4% of your OSD, then 9% difference in used space on your OSDs is caused by an OSD having only 2 more PGs than another OSD. If you do have very large PGs, increasing your PG count in those pools should improve your data distr

Re: [ceph-users] ceph balancer: further optimizations?

2018-08-20 Thread David Turner
i...@profihost.ag> wrote: > > Am 20.08.2018 um 22:13 schrieb David Turner: > > You might just have too much data per PG. If a single PG can account > > for 4% of your OSD, then 9% difference in used space on your OSDs is > > caused by an OSD having only 2 more PGs than anot

Re: [ceph-users] Network cluster / addr

2018-08-21 Thread David Turner
Private is only for OSDs. Nothing else communicates on that. MONs, MGRs, MDSs, RGWs, and clients all communicate on the public network. Even OSDs need to communicate with MONs on the public network. All of that said, it is generally considered useless to split your private and public subnets. Even

Re: [ceph-users] Upgrade to Infernalis: OSDs crash all the time

2018-08-21 Thread David Turner
Ceph does not support downgrading OSDs. When you removed the single OSD, it was probably trying to move data onto the other OSDs in the node with Infernalis OSDs. I would recommend stopping every OSD in that node and marking them out so the cluster will rebalance without them. Assuming your clus

Re: [ceph-users] Upgrade to Infernalis: OSDs crash all the time

2018-08-21 Thread David Turner
t; ones are crashing, not the new one. > Maybe with all the flags set (pause, norecover, ...) > > > Paul > > 2018-08-21 19:08 GMT+02:00 Kees Meijs : > > Hello David, > > > > Thank you and I'm terribly sorry; I was unaware I was starting new > threads. &g

Re: [ceph-users] There's a way to remove the block.db ?

2018-08-21 Thread David Turner
They have talked about working on allowing people to be able to do this, but for now there is nothing you can do to remove the block.db or block.wal from a bluestore OSD. However, there is an option to completely replace the SSD, not remove it. There are a few ML threads discussing how to utilize

Re: [ceph-users] HDD-only CephFS cluster with EC and without SSD/NVMe

2018-08-22 Thread David Turner
I would suggest having some flash media for their own OSDs to put the cephfs metadata pool onto. That was a pretty significant boost for me when I moved the metadata pool onto flash media. My home setup is only 3 nodes and is running EC 2+1 on pure HDD OSDs with metadata on SSDs. It's been runni

Re: [ceph-users] BlueStore options in ceph.conf not being used

2018-08-22 Thread David Turner
The config settings for DB and WAL size don't do anything. For journal sizes they would be used for creating your journal partition with ceph-disk, but ceph-volume does not use them for creating bluestore OSDs. You need to create the partitions for the DB and WAL yourself and supply those partitio

Re: [ceph-users] mgr/dashboard: backporting Ceph Dashboard v2 to Luminous

2018-08-22 Thread David Turner
My initial reaction to this PR/backport was questioning why such a major update would happen on a dot release of Luminous. Your reaction to keeping both dashboards viable goes to support that. Should we really be backporting features into a dot release that force people to change how they use the

Re: [ceph-users] BlueStore options in ceph.conf not being used

2018-08-22 Thread David Turner
Ceph will use all of this for > db/wal automatically? > > Thanks > R > > On Wed, Aug 22, 2018 at 2:09 PM Alfredo Deza wrote: > >> On Wed, Aug 22, 2018 at 2:48 PM, David Turner >> wrote: >> > The config settings for DB and WAL size don't do any

Re: [ceph-users] Unexpected behaviour after monitors upgrade from Jewel to Luminous

2018-08-22 Thread David Turner
I swear I remember a thread on the ML that talked about someone having increased memory usage on their OSDs after upgrading their MONs to Luminous as well, but I can't seem to find it. Iirc the problem for them was resolved when they finished the upgrade to Luminous. It might not be a bad idea fo

Re: [ceph-users] Ceph RGW Index Sharding In Jewel

2018-08-22 Thread David Turner
The release notes for 0.94.10 mention the introduction of the `radosgw-admin bucket reshard` command. Redhat [1] documentation for their Enterprise version of Jewel goes into detail for the procedure. You can also search the ML archives for the command to find several conversations about the proces

Re: [ceph-users] Can I deploy wal and db of more than one osd in one partition

2018-08-26 Thread David Turner
You need to do them on separate partitions. You can either do sdc{num} or manage the SSD using LVM. On Sun, Aug 26, 2018, 8:39 AM Zhenshi Zhou wrote: > Hi, > I have 4 osd nodes with 4 hdd and 1 ssd on each. > I'm gonna add these osds in an existing cluster. > What I'm confused is that how to dea

Re: [ceph-users] Odp.: pgs incomplete and inactive

2018-08-27 Thread David Turner
I came across a problem like this before with small flash OSDs for metadata. There is an open tracker about why it was able to fill 100% of the way up, but no work done on it in 6 months after I got back to healthy. The way I did that was deleting one copy of a PG from each OSD (different PGs on

Re: [ceph-users] Installing ceph 12.2.4 via Ubuntu apt

2018-08-28 Thread David Turner
That is the expected behavior of the ceph repo. In the past when I needed a specific version I would download the packages for the version to a folder and you can create a repo file that reads from a local directory. That's how I would re-install my test lab after testing an upgrade procedure to tr

Re: [ceph-users] Unrepairable PG

2018-08-28 Thread David Turner
There is a [1] tracker open for this issue. There are 2 steps that should get a pg to scrub/repair that is just issuing the scrub, but not running it. First is to increase osd_max_scrubs on the OSDs involved in the PG. If that doesn't fix it, then try increasing your osd_deep_scrub_interval on all

Re: [ceph-users] How to put ceph-fuse fstab remote path?

2018-08-28 Thread David Turner
This is what we have for our fstab to mount a specific subfolder using ceph-fuse id=cephfs-backup,client_mountpoint=/backup /home/backup2 fuse.ceph _netdev,noatime,rw 0 0 On Tue, Aug 28, 2018 at 4:04 AM Marc Roos wrote: > > kernel > c01,c02,c03:/backup /home/backupceph

Re: [ceph-users] cephfs mount on osd node

2018-08-29 Thread David Turner
The problem with mounting an RBD or CephFS on an OSD node is if you're doing so with the kernel client. In a previous message on the ML John Spray explained this wonderfully. "This is not a Ceph-specific thing -- it can also affect similar systems like Lustre. The classic case is when under so

Re: [ceph-users] Hammer and a (little) disk/partition shrink...

2018-08-29 Thread David Turner
Replace the raid controller in the chassis with an HBA before moving into the new hardware? ;) If you do move to the HP controller, make sure you're monitoring the health of the cache battery in the controller. We notice a significant increase to await on our OSD nodes behind these when the cache

[ceph-users] Mixed Bluestore and Filestore NVMe OSDs for RGW metadata both running out of space

2018-08-29 Thread David Turner
osd daemon perf dump for a one of my bluestore NVMe OSDs has [1] this excerpt. I grabbed those stats based on Wido's [2] script to determine how much DB overhead you have per object. My [3] calculations for this particular OSD are staggering. 99% of the space used on this OSD is being consumed b

Re: [ceph-users] safe to remove leftover bucket index objects

2018-08-30 Thread David Turner
I'm glad you asked this, because it was on my to-do list. I know that based on our not existing in the bucket marker does not mean it's safe to delete. I have an index pool with 22k objects in it. 70 objects match existing bucket markers. I was having a problem on the cluster and started deleting

Re: [ceph-users] mimic/bluestore cluster can't allocate space for bluefs

2018-08-30 Thread David Turner
I have 2 OSDs failing to start due to this [1] segfault. What is happening matches what Sage said about this [2] bug. The OSDs are NVMe disks and rocksdb is compacting omaps. I attempted setting `bluestore_bluefs_min_free = 10737418240` and then start the OSDs, but they both segfaulted with the

Re: [ceph-users] rocksdb mon stores growing until restart

2018-08-30 Thread David Turner
The Hammer ticket was https://tracker.ceph.com/issues/13990. The problem here was when OSDs asked each other for which map they needed to keep and a leak would set it to NULL then that OSD would never delete an OSD map again until it was restarted. On Thu, Aug 30, 2018 at 3:09 AM Joao Eduardo Lui

[ceph-users] Luminous missing osd_backfill_full_ratio

2018-08-30 Thread David C
Hi All I feel like this is going to be a silly query with a hopefully simple answer. I don't seem to have the osd_backfill_full_ratio config option on my OSDs and can't inject it. This a Lumimous 12.2.1 cluster that was upgraded from Jewel. I added an OSD to the cluster and woke up the next day t

Re: [ceph-users] Luminous missing osd_backfill_full_ratio

2018-08-30 Thread David Turner
This moved to the PG map in luminous. I think it might have been there in Jewel as well. http://docs.ceph.com/docs/luminous/man/8/ceph/#pg ceph pg set_full_ratio ceph pg set_backfillfull_ratio ceph pg set_nearfull_ratio On Thu, Aug 30, 2018, 1:57 PM David C wrote: > Hi All > > I

Re: [ceph-users] Best practices for allocating memory to bluestore cache

2018-08-30 Thread David Turner
Be very careful trying to utilize more RAM while your cluster is healthy. When you're going to need the extra RAM is when you're closer is unhealthy and your osds are peering, recovering, backfilling, etc. That's when the osd daemons start needing the RAM that is recommended in the docs. On Thu, A

Re: [ceph-users] filestore split settings

2018-08-31 Thread David Turner
More important than being able to push those settings or further is probably the ability to actually split your subfolders. I've been using variants of this [1] script I created a while back to take care of that. To answer your question, we do run with much larger settings than you're using. 128/-

Re: [ceph-users] cephfs speed

2018-08-31 Thread David Byte
Are these single threaded writes that you are referring to? It certainly appears so from the thread, but I thought it would be good to confirm that before digging in further. David Byte Sr. Technology Strategist SCE Enterprise Linux SCE Enterprise Storage Alliances and SUSE Embedded db

Re: [ceph-users] Adding node efficient data move.

2018-09-01 Thread David Turner
I've heard you can do that with the manager service for balancing your cluster. You can set the maximum amount of misplaced objects you want and the service will add in the new node until it's balanced without moving more data than your settings specify at any time. On Sat, Sep 1, 2018, 6:35 AM Ma

[ceph-users] "no valid command found" when running "ceph-deploy osd create"

2018-09-01 Thread David Wahler
python-rgw 13.2.1-1 arm64Python 2 libraries for the Ceph librgw library Thanks, -- David ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Help Basically..

2018-09-02 Thread David C
Does "ceph health detail" work? Have you manually confirmed the OSDs on the nodes are working? What was the replica size of the pools? Are you seeing any progress with the recovery? On Sun, Sep 2, 2018 at 9:42 AM Lee wrote: > Running 0.94.5 as part of a Openstack enviroment, our ceph setup is

Re: [ceph-users] Help Basically..

2018-09-02 Thread David Turner
When the first node went offline with a dead SSD journal, all of the dates on the OSDs was useless. Unless you could flush the journals, you can't guarantee that a wire the cluster think happened actually made it to the disk. The proper procedure here is to remove those OSDs and add them again as

Re: [ceph-users] Help Basically..

2018-09-02 Thread David Turner
://www.sebastien-han.fr/blog/2014/11/27/ceph-recover-osds-after-ssd-journal-failure/ > > Which they all started without a problem. > > > On Sun, 2 Sep 2018 at 15:43, David Turner wrote: > >> It looks like osds on the first failed node are having problems. What >> com

Re: [ceph-users] "no valid command found" when running "ceph-deploy osd create"

2018-09-02 Thread David Wahler
ceph-deploy logs are a bit confusing. I submitted a PR to add a brief note to the quick-start guide, in case anyone else makes the same mistake: https://github.com/ceph/ceph/pull/23879 Thanks for the assistance! -- David On Sun, Sep 2, 2018 at 7:44 AM Alfredo Deza wrote: > > There shoul

Re: [ceph-users] Help Basically..

2018-09-02 Thread David Turner
Agreed on not going the disks until your cluster is healthy again. Making them out and seeing how healthy you can get in the meantime is a good idea. On Sun, Sep 2, 2018, 1:18 PM Ronny Aasen wrote: > On 02.09.2018 17:12, Lee wrote: > > Should I just out the OSD's first or completely zap them and

Re: [ceph-users] "no valid command found" when running "ceph-deploy osd create"

2018-09-02 Thread David Wahler
On Sun, Sep 2, 2018 at 1:31 PM Alfredo Deza wrote: > > On Sun, Sep 2, 2018 at 12:00 PM, David Wahler wrote: > > Ah, ceph-volume.log pointed out the actual problem: > > > > RuntimeError: Cannot use device (/dev/storage/bluestore). A vg/lv path > > or an existing de

Re: [ceph-users] Luminous missing osd_backfill_full_ratio

2018-09-03 Thread David C
e if you delay the "...require-osd-release luminous" for whatever reason as it appears to leave you with no backfillfull limit Still having a bit of an issue with new OSDs over filling but will start a new thread for that Cheers, On Thu, Aug 30, 2018 at 10:34 PM David Turner wrote: &

[ceph-users] Luminous new OSD being over filled

2018-09-03 Thread David C
full? Do I just have to leave this one with a lower crush weight and then continue adding the drives and then eventually even out the crush weights? Thanks David ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Luminous new OSD being over filled

2018-09-03 Thread David C
h osd crush reweight osd.27 2 > sudo -u ceph ceph osd crush reweight osd.28 2 > sudo -u ceph ceph osd crush reweight osd.29 2 > > Etc etc > > > -Original Message- > From: David C [mailto:dcsysengin...@gmail.com] > Sent: maandag 3 september 2018 14:34 > To: ce

Re: [ceph-users] Ceph Luminous - journal setting

2018-09-04 Thread David Turner
Are you planning on using bluestore or filestore? The settings for filestore haven't changed. If you're planning to use bluestore there is a lot of documentation in the ceph docs as well as a wide history of questions like this on the ML. On Mon, Sep 3, 2018 at 5:24 AM M Ranga Swami Reddy wrote

Re: [ceph-users] Luminous RGW errors at start

2018-09-04 Thread David Turner
I was confused what could be causing this until Janne's email. I think they're correct that the cluster is preventing pool creation due to too many PGs per OSD. Double check how many PGs you have in each pool and what your defaults are for that. On Mon, Sep 3, 2018 at 7:19 AM Janne Johansson wr

Re: [ceph-users] Luminous new OSD being over filled

2018-09-04 Thread David Turner
preventing your cluster from being poorly balanced. On Mon, Sep 3, 2018 at 12:08 PM David C wrote: > Hi Marc > > I like that approach although I think I'd go in smaller weight increments. > > Still a bit confused by the behaviour I'm seeing, it looks like I've got > th

Re: [ceph-users] v12.2.8 Luminous released

2018-09-05 Thread David Turner
> - > > * bluestore: set correctly shard for existed Collection (issue#24761, > pr#22860, Jianpeng Ma) > > * build/ops: Boost system library is no longer required to compile and > link example librados program (issue#25054, pr#23202, Nathan Cutler) > > * build/op

Re: [ceph-users] [Ceph-community] How to setup Ceph OSD auto boot up on node reboot

2018-09-05 Thread David Turner
The magic sauce to get Filestore OSDs to start on a node reboot is to make sure that all of your udev magic is correct. In particular you need to have the correct UUID set for all partitions. I haven't dealt with it in a long time, but I've written up a few good ML responses about it. On Tue, Se

Re: [ceph-users] help needed

2018-09-06 Thread David Turner
The official ceph documentation recommendations for a db partition for a 4TB bluestore osd would be 160GB each. Samsung Evo Pro is not an Enterprise class SSD. A quick search of the ML will allow which SSDs people are using. As was already suggested, the better option is an HBA as opposed to a ra

Re: [ceph-users] Ceph talks from Mounpoint.io

2018-09-06 Thread David Turner
They mentioned that they were going to send the slides to everyone that had their badges scanned at the conference. I haven't seen that email come out yet, though. On Thu, Sep 6, 2018 at 4:14 PM Gregory Farnum wrote: > Unfortunately I don't believe anybody collected the slide files, so they > a

[ceph-users] CephFS tar archiving immediately after writing

2018-09-07 Thread David Turner
We have an existing workflow that we've moved from one server sharing a local disk via NFS to secondary servers to all of them mounting CephFS. The primary server runs a script similar to [1] this, but since we've moved it into CephFS, we get [2] this error. We added the sync in there to try to he

Re: [ceph-users] CephFS tar archiving immediately after writing

2018-09-07 Thread David Turner
ush of those rstats up the > tree to enable good differential backups. Not sure what the status of that > is. > -Greg > > On Fri, Sep 7, 2018 at 11:06 AM David Turner > wrote: > >> We have an existing workflow that we've moved from one server sharing a >> local

Re: [ceph-users] CephFS tar archiving immediately after writing

2018-09-07 Thread David Turner
intrinsic to other necessary functions) so you can't disable it on the > server. > -Greg > > On Fri, Sep 7, 2018 at 11:48 AM David Turner > wrote: > >> Is it be possible to disable this feature? Very few filesystems >> calculate the size of its folder's

Re: [ceph-users] advice with erasure coding

2018-09-08 Thread David Turner
I tested running VMs on EC back in Hammer. The performance was just bad. I didn't even need much io, but even performing standard maintenance was annoying enough that I abandoned the idea. I didn't really try to tweak settings to make it work and I only had a 3 node cluster running 2+1. I did use i

Re: [ceph-users] Mimic upgrade failure

2018-09-08 Thread David Turner
What osd/mon/etc config settings do you have that are not default? It might be worth utilizing nodown to stop osds from marking each other down and finish the upgrade to be able to set the minimum osd version to mimic. Stop the osds in a node, manually mark them down, start them back up in mimic. D

<    4   5   6   7   8   9   10   11   12   13   >