Re: [ceph-users] Ceph mirrors
Quoting Wido den Hollander (w...@42on.com):
> > I want to suggest that we keep the older packages in the repo list.
> > They are on the mirrors anyway (../debian/pool/main/{c,r}/ceph/).
> They still are, aren't they?
> http://eu.ceph.com/debian-luminous/pool/main/c/ceph/
Well, yes. But the repository's metadata doesn't list them.
http://eu.ceph.com/debian-luminous/dists/xenial/main/binary-amd64/Packages
only lists 12.2.1-1xenial packages, the 12.{0,1}.x packages aren't
listed which makes you can't "apt install ceph-base=12.0.1-1xenial"
should you want to do that for some reason.
Pinning, or essentially "hold"-ing packages on a certain version will
still work, but it's not 'as easy' to downgrade right now.
My 25ct,
-Sndr.
--
| Getting dressed in clothes you buy for work, driving through traffic
| in a car you are still paying for in order to get to the job you
| need to pay for the clothes, the car and the house you leave vacant
| all day so you can afford to live in it.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] What's about release-note for 10.2.10?
Hi, again is an update available without release-note... http://ceph.com/releases/v10-2-10-jewel-released/ isn't found. No announcement on the mailing list (perhaps i miss something). I know, normaly it's save to update ceph, but two releases ago it wasn't. Udo ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph mgr influx module on luminous
On Thu, Oct 5, 2017 at 11:55 PM, Stefan Kooman wrote:
> Hi,
>
> The influx module had made it's appearance in Mimic. I'm eager to try it out
> on
> luminous. So I went ahead and put the code [1] in /usr/lib/ceph/mgr/influx.
>
> I installed "python-influxdb" because it's a dependency (Ubuntu 16.0.4.3):
>
> Python 2.7.12 (default, Nov 19 2016, 06:48:10)
> [GCC 5.4.0 20160609] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> from influxdb import InfluxDBClient
> >>> print InfluxDBClient
>
> >>>
>
> Activating the module:
>
> ceph mgr module enable influx
>
> Gives me (ceph-mgr log):
>
> 2017-10-06 00:38:24.183802 7f0f98fcc700 -1 mgr got_mgr_map mgrmap module list
> changed to (dashboard,influx,restful,status), respawn
> 2017-10-06 00:38:24.202465 7f9f57e6d500 10 mgr send_beacon sending beacon as
> gid 2642816 modules dashboard,influx,prometheus,restful,status,zabbix
> 2017-10-06 00:38:25.256103 7f9f4b9da700 1 mgr init Loading python module
> 'influx'
> 2017-10-06 00:38:25.259090 7f9f4b9da700 -1 mgr load Module not found: 'influx'
> File "/usr/lib/ceph/mgr/influx/__init__.py", line 1, in
> IndentationError: ('unexpected indent',
> ('/usr/lib/ceph/mgr/influx/module.py', 10, 4, 'from influxdb import
> InfluxDBClient\n'))
> 2017-10-06 00:38:25.259170 7f9f4b9da700 -1 mgr init Error loading module
> 'influx': (2) No such file or directory
> 2017-10-06 00:38:25.346411 7f9f4b9da700 -1 log_channel(cluster) log [ERR] :
> Failed to load ceph-mgr modules: influx
If you're getting an IndentationError then my guess would be that
you've gained a typo while copy-pasting something.
> From above I understand the manager cannot find the module. What else do I
> need
> to do to activate the influx module (in luminous)?
ceph-mgr modules are not generally transplantable between Ceph
versions (although they may work sometimes) -- I do intend to backport
this module to luminous in a forthcoming 12.2.x though.
John
>
> Thanks,
>
> Gr. Stefan
>
> [1]: https://github.com/ceph/ceph/blob/master/src/pybind/mgr/influx/
>
> --
> | BIT BV http://www.bit.nl/Kamer van Koophandel 09090351
> | GPG: 0xD14839C6 +31 318 648 688 / i...@bit.nl
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] What's about release-note for 10.2.10?
On 17-10-06 11:25, ulem...@polarzone.de wrote: Hi, again is an update available without release-note... http://ceph.com/releases/v10-2-10-jewel-released/ isn't found. No announcement on the mailing list (perhaps i miss something). While I do not see v10.2.10 tag in repo yet, it looks like packages were built less than two days ago. So maybe they didn't have time to send release notes yet? Building takes time and sending notes before packages are available is not a good idea too. I know, normaly it's save to update ceph, but two releases ago it wasn't. Udo ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph cache pool full
On Fri, Oct 6, 2017, 1:05 AM Christian Balzer wrote: > > Hello, > > On Fri, 06 Oct 2017 03:30:41 + David Turner wrote: > > > You're missing most all of the important bits. What the osds in your > > cluster look like, your tree, and your cache pool settings. > > > > ceph df > > ceph osd df > > ceph osd tree > > ceph osd pool get cephfs_cache all > > > Especially the last one. > > My money is on not having set target_max_objects and target_max_bytes to > sensible values along with the ratios. > In short, not having read the (albeit spotty) documentation. > > > You have your writeback cache on 3 nvme drives. It looks like you have > > 1.6TB available between them for the cache. I don't know the behavior of > a > > writeback cache tier on cephfs for large files, but I would guess that it > > can only hold full files and not flush partial files. > > I VERY much doubt that, if so it would be a massive flaw. > One assumes that cache operations work on the RADOS object level, no > matter what. > I hope that it is on the rados level, but not a single object had been flushed to the backing pool. So I hazarded a guess. Seeing his settings will shed more light. > > > That would mean your > > cache needs to have enough space for any file being written to the > cluster. > > In this case a 1.3TB file with 3x replication would require 3.9TB (more > > than double what you have available) of available space in your writeback > > cache. > > > > There are very few use cases that benefit from a cache tier. The docs for > > Luminous warn as much. > You keep repeating that like a broken record. > > And while certainly not false I for one wouldn't be able to use (justify > using) Ceph w/o cache tiers in our main use case. > In this case I assume they were following and old cheat sheet or such, > suggesting the previously required cache tier with EC pools. > http://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/ I know I keep repeating it, especially recently as there have been a lot of people asking about it. The Luminous docs added a large section about how it is probably not what you want. Like me, it is not saying that there are no use cases for it. There was no information provided about the use case and I made some suggestions/guesses. I'm also guessing that they are following a guide where a writeback cache was necessary for CephFS to use EC prior to Luminous. I also usually add that people should test it out and find what works best for them. I will always defer to your practical use of cache tiers as well, especially when using rbds. I manage a cluster that I intend to continue running a writeback cache in front of CephFS on the same drives as the EC pool. The use case receives a good enough benefit from the cache tier that it isn't even required to use flash media to see it. It is used for video editing and the files are usually modified and read within the first 24 hours and then left in cold storage until deleted. I have the cache timed to keep everything in it for 24 hours and then evict it by using a minimum time to flush and evict at 24 hours and a target max bytes of 0. All files are in there for that time and then it never has to decide what to keep as it doesn't keep anything longer than that. Luckily read performance from cold storage is not a requirement of this cluster as any read operation has to first read it from EC storage, write it to replica storage, and then read it from replica storage... Yuck. > > Christian > > >What is your goal by implementing this cache? If the > > answer is to utilize extra space on the nvmes, then just remove it and > say > > thank you. The better use of nvmes in that case are as a part of the > > bluestore stack and give your osds larger DB partitions. Keeping your > > metadata pool on nvmes is still a good idea. > > > > On Thu, Oct 5, 2017, 7:45 PM Shawfeng Dong wrote: > > > > > Dear all, > > > > > > We just set up a Ceph cluster, running the latest stable release Ceph > > > v12.2.0 (Luminous): > > > # ceph --version > > > ceph version 12.2.0 (32ce2a3ae5239ee33d6150705cdb24d43bab910c) luminous > > > (rc) > > > > > > The goal is to serve Ceph filesystem, for which we created 3 pools: > > > # ceph osd lspools > > > 1 cephfs_data,2 cephfs_metadata,3 cephfs_cache, > > > where > > > * cephfs_data is the data pool (36 OSDs on HDDs), which is > erased-coded; > > > * cephfs_metadata is the metadata pool > > > * cephfs_cache is the cache tier (3 OSDs on NVMes) for cephfs_data. The > > > cache-mode is writeback. > > > > > > Everything had worked fine, until today when we tried to copy a 1.3TB > file > > > to the CephFS. We got the "No space left on device" error! > > > > > > 'ceph -s' says some OSDs are full: > > > # ceph -s > > > cluster: > > > id: e18516bf-39cb-4670-9f13-88ccb7d19769 > > > health: HEALTH_ERR > > > full flag(s) set > > > 1 full osd(s) > > > 1 pools have many more objects per pg than average > >
[ceph-users] Re : Re : Re : bad crc/signature errors
Le jeudi 05 octobre 2017 à 21:52 +0200, Ilya Dryomov a écrit :
> On Thu, Oct 5, 2017 at 6:05 PM, Olivier Bonvalet > wrote:
> > Le jeudi 05 octobre 2017 à 17:03 +0200, Ilya Dryomov a écrit :
> > > When did you start seeing these errors? Can you correlate that
> > > to
> > > a ceph or kernel upgrade? If not, and if you don't see other
> > > issues,
> > > I'd write it off as faulty hardware.
> >
> > Well... I have one hypervisor (Xen 4.6 and kernel Linux 4.1.13),
> > which
>
> Is that 4.1.13 or 4.13.1?
>
Linux 4.1.13. The old Debian 8, with Xen 4.6 from upstream.
> > have the problem for a long time, at least since 1 month (I haven't
> > older logs).
> >
> > But, on others hypervisors (Xen 4.8 with Linux 4.9.x), I haven't
> > the
> > problem.
> > And it's when I upgraded thoses hypervisors to Linux 4.13.x, that
> > "bad
> > crc" errors appeared.
> >
> > Note : if I upgraded kernels on Xen 4.8 hypervisors, it's because
> > some
> > DISCARD commands over RBD were blocking ("fstrim" works, but not
> > "lvremove" with discard enabled). After upgrading to Linux 4.13.3,
> > DISCARD works again on Xen 4.8.
>
> Which kernel did you upgrade from to 4.13.3 exactly?
>
>
4.9.47 or 4.9.52, I don't have more precise data about this.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph cache pool full
Dear all, Thanks a lot for the very insightful comments/suggestions! There are 3 OSD servers in our pilot Ceph cluster, each with 2x 1TB SSDs (boot disks), 12x 8TB SATA HDDs and 2x 1.2TB NVMe SSDs. We use the bluestore backend, with the first NVMe as the WAL and DB devices for OSDs on the HDDs. And we try to create a cache tier out of the second NVMes. Here are the outputs of the commands suggested by David: 1) # ceph df GLOBAL: SIZE AVAIL RAW USED %RAW USED 265T 262T2847G 1.05 POOLS: NAMEID USED %USED MAX AVAIL OBJECTS cephfs_data 1 0 0 248T 0 cephfs_metadata 2 8515k 0 248T 24 cephfs_cache3 1381G 100.00 0 355385 2) # ceph osd df 0 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 174 1 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 169 2 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 173 3 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 159 4 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 173 5 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 162 6 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 149 7 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 179 8 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 163 9 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 194 10 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 185 11 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 168 36 nvme 1.09149 1.0 1117G 855G 262G 76.53 73.01 79 12 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 180 13 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 168 14 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 178 15 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 170 16 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 149 17 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 203 18 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 173 19 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 158 20 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 154 21 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 160 22 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 167 23 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 188 37 nvme 1.09149 1.0 1117G 1061G 57214M 95.00 90.63 98 24 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 187 25 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 200 26 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 147 27 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 171 28 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 162 29 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 152 30 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 174 31 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 176 32 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 182 33 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 155 34 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 166 35 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 176 38 nvme 1.09149 1.0 1117G 857G 260G 76.71 73.18 79 TOTAL 265T 2847G 262T 1.05 MIN/MAX VAR: 0.03/90.63 STDDEV: 22.81 3) # ceph osd tree -1 265.29291 root default -388.43097 host pulpo-osd01 0 hdd 7.27829 osd.0up 1.0 1.0 1 hdd 7.27829 osd.1up 1.0 1.0 2 hdd 7.27829 osd.2up 1.0 1.0 3 hdd 7.27829 osd.3up 1.0 1.0 4 hdd 7.27829 osd.4up 1.0 1.0 5 hdd 7.27829 osd.5up 1.0 1.0 6 hdd 7.27829 osd.6up 1.0 1.0 7 hdd 7.27829 osd.7up 1.0 1.0 8 hdd 7.27829 osd.8up 1.0 1.0 9 hdd 7.27829 osd.9up 1.0 1.0 10 hdd 7.27829 osd.10 up 1.0 1.0 11 hdd 7.27829 osd.11 up 1.0 1.0 36 nvme 1.09149 osd.36 up 1.0 1.0 -588.43097 host pulpo-osd02 12 hdd 7.27829 osd.12 up 1.0 1.0 13 hdd 7.27829 osd.13 up 1.0 1.0 14 hdd 7.27829 osd.14 up 1.0 1.0 15 hdd 7.27829 osd.15 up 1.0 1.0 16 hdd 7.27829 osd.16 up 1.0 1.0 17 hdd 7.27829 osd.17 up 1.0 1.0 18 hdd 7.27829 osd.18 up 1.0 1.0 19 hdd 7.27829 osd.19 up 1.0 1.0 20 hdd 7.27829 osd.20 up 1.0 1.0 21 hdd 7.27829 osd.21 up 1.0 1.0 22 hdd 7.27829 osd.22 up 1.0 1.0 23 hdd 7
Re: [ceph-users] Ceph cache pool full
Not looking at anything else, you didn't set the max_bytes or max_objects for it to start flushing... On Fri, Oct 6, 2017 at 4:49 PM, Shawfeng Dong wrote: > Dear all, > > Thanks a lot for the very insightful comments/suggestions! > > There are 3 OSD servers in our pilot Ceph cluster, each with 2x 1TB SSDs > (boot disks), 12x 8TB SATA HDDs and 2x 1.2TB NVMe SSDs. We use the bluestore > backend, with the first NVMe as the WAL and DB devices for OSDs on the HDDs. > And we try to create a cache tier out of the second NVMes. > > Here are the outputs of the commands suggested by David: > > 1) # ceph df > GLOBAL: > SIZE AVAIL RAW USED %RAW USED > 265T 262T2847G 1.05 > POOLS: > NAMEID USED %USED MAX AVAIL OBJECTS > cephfs_data 1 0 0 248T 0 > cephfs_metadata 2 8515k 0 248T 24 > cephfs_cache3 1381G 100.00 0 355385 > > 2) # ceph osd df > 0 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 174 > 1 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 169 > 2 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 173 > 3 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 159 > 4 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 173 > 5 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 162 > 6 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 149 > 7 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 179 > 8 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 163 > 9 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 194 > 10 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 185 > 11 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 168 > 36 nvme 1.09149 1.0 1117G 855G 262G 76.53 73.01 79 > 12 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 180 > 13 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 168 > 14 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 178 > 15 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 170 > 16 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 149 > 17 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 203 > 18 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 173 > 19 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 158 > 20 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 154 > 21 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 160 > 22 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 167 > 23 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 188 > 37 nvme 1.09149 1.0 1117G 1061G 57214M 95.00 90.63 98 > 24 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 187 > 25 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 200 > 26 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 147 > 27 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 171 > 28 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 162 > 29 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 152 > 30 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 174 > 31 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 176 > 32 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 182 > 33 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 155 > 34 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 166 > 35 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 176 > 38 nvme 1.09149 1.0 1117G 857G 260G 76.71 73.18 79 > TOTAL 265T 2847G 262T 1.05 > MIN/MAX VAR: 0.03/90.63 STDDEV: 22.81 > > 3) # ceph osd tree > -1 265.29291 root default > -388.43097 host pulpo-osd01 > 0 hdd 7.27829 osd.0up 1.0 1.0 > 1 hdd 7.27829 osd.1up 1.0 1.0 > 2 hdd 7.27829 osd.2up 1.0 1.0 > 3 hdd 7.27829 osd.3up 1.0 1.0 > 4 hdd 7.27829 osd.4up 1.0 1.0 > 5 hdd 7.27829 osd.5up 1.0 1.0 > 6 hdd 7.27829 osd.6up 1.0 1.0 > 7 hdd 7.27829 osd.7up 1.0 1.0 > 8 hdd 7.27829 osd.8up 1.0 1.0 > 9 hdd 7.27829 osd.9up 1.0 1.0 > 10 hdd 7.27829 osd.10 up 1.0 1.0 > 11 hdd 7.27829 osd.11 up 1.0 1.0 > 36 nvme 1.09149 osd.36 up 1.0 1.0 > -588.43097 host pulpo-osd02 > 12 hdd 7.27829 osd.12 up 1.0 1.0 > 13 hdd 7.27829 osd.13 up 1.0 1.0 > 14 hdd 7.27829 osd.14 up 1.0 1.0 > 15 hdd 7.27829 osd.15 up 1.0 1.0 > 16 hdd 7.27829 osd.16 up 1.0 1.0 > 17 hdd 7.27829 osd.17 up 1.0 1.0 > 18
[ceph-users] "ceph osd status" fails
When I try to run the command "ceph osd status" on my cluster, I just get an error. Luckily unlike the last issue I had with ceph fs commands it doesn't seem to be crashing any of the daemons. root@vm-ds-01:/var/log/ceph# ceph osd status Error EINVAL: Traceback (most recent call last): File "/usr/lib/ceph/mgr/status/module.py", line 293, in handle_command return self.handle_osd_status(cmd) File "/usr/lib/ceph/mgr/status/module.py", line 273, in handle_osd_status stats = osd_stats[osd_id] KeyError: (78L,) Example and relevant excerpt from the ceph-mgr log shown at https://gist.github.com/rjhesketh/378ec118e42289a2dd0b1dd2462aae92 Is this trying to poll stats for an OSD which doesn't exist and therefore breaking? Rich signature.asc Description: OpenPGP digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph cache pool full
Thanks, Luis. I've just set max_bytes and max_objects: target_max_objects: 100 (1M) target_max_bytes: 1099511627776 (1TB) but nothing appears to be happening. Is there a way to force flushing? Thanks, Shaw On Fri, Oct 6, 2017 at 8:55 AM, Luis Periquito wrote: > Not looking at anything else, you didn't set the max_bytes or > max_objects for it to start flushing... > > On Fri, Oct 6, 2017 at 4:49 PM, Shawfeng Dong wrote: > > Dear all, > > > > Thanks a lot for the very insightful comments/suggestions! > > > > There are 3 OSD servers in our pilot Ceph cluster, each with 2x 1TB SSDs > > (boot disks), 12x 8TB SATA HDDs and 2x 1.2TB NVMe SSDs. We use the > bluestore > > backend, with the first NVMe as the WAL and DB devices for OSDs on the > HDDs. > > And we try to create a cache tier out of the second NVMes. > > > > Here are the outputs of the commands suggested by David: > > > > 1) # ceph df > > GLOBAL: > > SIZE AVAIL RAW USED %RAW USED > > 265T 262T2847G 1.05 > > POOLS: > > NAMEID USED %USED MAX AVAIL OBJECTS > > cephfs_data 1 0 0 248T 0 > > cephfs_metadata 2 8515k 0 248T 24 > > cephfs_cache3 1381G 100.00 0 355385 > > > > 2) # ceph osd df > > 0 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 174 > > 1 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 169 > > 2 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 173 > > 3 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 159 > > 4 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 173 > > 5 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 162 > > 6 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 149 > > 7 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 179 > > 8 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 163 > > 9 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 194 > > 10 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 185 > > 11 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 168 > > 36 nvme 1.09149 1.0 1117G 855G 262G 76.53 73.01 79 > > 12 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 180 > > 13 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 168 > > 14 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 178 > > 15 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 170 > > 16 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 149 > > 17 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 203 > > 18 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 173 > > 19 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 158 > > 20 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 154 > > 21 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 160 > > 22 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 167 > > 23 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 188 > > 37 nvme 1.09149 1.0 1117G 1061G 57214M 95.00 90.63 98 > > 24 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 187 > > 25 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 200 > > 26 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 147 > > 27 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 171 > > 28 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 162 > > 29 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 152 > > 30 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 174 > > 31 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 176 > > 32 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 182 > > 33 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 155 > > 34 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 166 > > 35 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 176 > > 38 nvme 1.09149 1.0 1117G 857G 260G 76.71 73.18 79 > > TOTAL 265T 2847G 262T 1.05 > > MIN/MAX VAR: 0.03/90.63 STDDEV: 22.81 > > > > 3) # ceph osd tree > > -1 265.29291 root default > > -388.43097 host pulpo-osd01 > > 0 hdd 7.27829 osd.0up 1.0 1.0 > > 1 hdd 7.27829 osd.1up 1.0 1.0 > > 2 hdd 7.27829 osd.2up 1.0 1.0 > > 3 hdd 7.27829 osd.3up 1.0 1.0 > > 4 hdd 7.27829 osd.4up 1.0 1.0 > > 5 hdd 7.27829 osd.5up 1.0 1.0 > > 6 hdd 7.27829 osd.6up 1.0 1.0 > > 7 hdd 7.27829 osd.7up 1.0 1.0 > > 8 hdd 7.27829 osd.8up 1.0 1.0 > > 9 hdd 7.27829 osd.9up 1.0 1.0 > > 10 hdd 7.27829 osd.10 up 1.0 1.0 > > 11 hdd 7.27829 osd.11 up 1.0 1.0 > > 36 nvme 1.09149 osd.36 up 1.0 1.0 > >
Re: [ceph-users] Ceph cache pool full
On Fri, 6 Oct 2017 16:55:31 +0100 Luis Periquito wrote: > Not looking at anything else, you didn't set the max_bytes or > max_objects for it to start flushing... > Precisely! He says, cackling, as he goes to cash in his bet. ^o^ > On Fri, Oct 6, 2017 at 4:49 PM, Shawfeng Dong wrote: > > Dear all, > > > > Thanks a lot for the very insightful comments/suggestions! > > > > There are 3 OSD servers in our pilot Ceph cluster, each with 2x 1TB SSDs > > (boot disks), 12x 8TB SATA HDDs and 2x 1.2TB NVMe SSDs. We use the bluestore > > backend, with the first NVMe as the WAL and DB devices for OSDs on the HDDs. > > And we try to create a cache tier out of the second NVMes. > > > > Here are the outputs of the commands suggested by David: > > > > 1) # ceph df > > GLOBAL: > > SIZE AVAIL RAW USED %RAW USED > > 265T 262T2847G 1.05 > > POOLS: > > NAMEID USED %USED MAX AVAIL OBJECTS > > cephfs_data 1 0 0 248T 0 > > cephfs_metadata 2 8515k 0 248T 24 > > cephfs_cache3 1381G 100.00 0 355385 > > > > 2) # ceph osd df > > 0 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 174 > > 1 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 169 > > 2 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 173 > > 3 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 159 > > 4 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 173 > > 5 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 162 > > 6 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 149 > > 7 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 179 > > 8 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 163 > > 9 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 194 > > 10 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 185 > > 11 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 168 > > 36 nvme 1.09149 1.0 1117G 855G 262G 76.53 73.01 79 > > 12 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 180 > > 13 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 168 > > 14 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 178 > > 15 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 170 > > 16 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 149 > > 17 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 203 > > 18 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 173 > > 19 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 158 > > 20 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 154 > > 21 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 160 > > 22 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 167 > > 23 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 188 > > 37 nvme 1.09149 1.0 1117G 1061G 57214M 95.00 90.63 98 > > 24 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 187 > > 25 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 200 > > 26 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 147 > > 27 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 171 > > 28 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 162 > > 29 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 152 > > 30 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 174 > > 31 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 176 > > 32 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 182 > > 33 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 155 > > 34 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 166 > > 35 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 176 > > 38 nvme 1.09149 1.0 1117G 857G 260G 76.71 73.18 79 > > TOTAL 265T 2847G 262T 1.05 > > MIN/MAX VAR: 0.03/90.63 STDDEV: 22.81 > > > > 3) # ceph osd tree > > -1 265.29291 root default > > -388.43097 host pulpo-osd01 > > 0 hdd 7.27829 osd.0up 1.0 1.0 > > 1 hdd 7.27829 osd.1up 1.0 1.0 > > 2 hdd 7.27829 osd.2up 1.0 1.0 > > 3 hdd 7.27829 osd.3up 1.0 1.0 > > 4 hdd 7.27829 osd.4up 1.0 1.0 > > 5 hdd 7.27829 osd.5up 1.0 1.0 > > 6 hdd 7.27829 osd.6up 1.0 1.0 > > 7 hdd 7.27829 osd.7up 1.0 1.0 > > 8 hdd 7.27829 osd.8up 1.0 1.0 > > 9 hdd 7.27829 osd.9up 1.0 1.0 > > 10 hdd 7.27829 osd.10 up 1.0 1.0 > > 11 hdd 7.27829 osd.11 up 1.0 1.0 > > 36 nvme 1.09149 osd.36 up 1.0 1.0 > > -588.43097 host pulpo-osd02 > > 12 hdd 7.27829 osd.12 up 1.0 1.0 > > 13 hdd 7.27829 osd.13
Re: [ceph-users] Ceph cache pool full
I found the command: rados -p cephfs_cache cache-flush-evict-all The documentation ( http://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/) has been improved a lot since I last checked it a few weeks ago! -Shaw On Fri, Oct 6, 2017 at 9:10 AM, Shawfeng Dong wrote: > Thanks, Luis. > > I've just set max_bytes and max_objects: > target_max_objects: 100 (1M) > target_max_bytes: 1099511627776 (1TB) > > but nothing appears to be happening. Is there a way to force flushing? > > Thanks, > Shaw > > On Fri, Oct 6, 2017 at 8:55 AM, Luis Periquito > wrote: > >> Not looking at anything else, you didn't set the max_bytes or >> max_objects for it to start flushing... >> >> On Fri, Oct 6, 2017 at 4:49 PM, Shawfeng Dong wrote: >> > Dear all, >> > >> > Thanks a lot for the very insightful comments/suggestions! >> > >> > There are 3 OSD servers in our pilot Ceph cluster, each with 2x 1TB SSDs >> > (boot disks), 12x 8TB SATA HDDs and 2x 1.2TB NVMe SSDs. We use the >> bluestore >> > backend, with the first NVMe as the WAL and DB devices for OSDs on the >> HDDs. >> > And we try to create a cache tier out of the second NVMes. >> > >> > Here are the outputs of the commands suggested by David: >> > >> > 1) # ceph df >> > GLOBAL: >> > SIZE AVAIL RAW USED %RAW USED >> > 265T 262T2847G 1.05 >> > POOLS: >> > NAMEID USED %USED MAX AVAIL >> OBJECTS >> > cephfs_data 1 0 0 248T >> 0 >> > cephfs_metadata 2 8515k 0 248T >> 24 >> > cephfs_cache3 1381G 100.00 0 >> 355385 >> > >> > 2) # ceph osd df >> > 0 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 174 >> > 1 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 169 >> > 2 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 173 >> > 3 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 159 >> > 4 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 173 >> > 5 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 162 >> > 6 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 149 >> > 7 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 179 >> > 8 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 163 >> > 9 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 194 >> > 10 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 185 >> > 11 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 168 >> > 36 nvme 1.09149 1.0 1117G 855G 262G 76.53 73.01 79 >> > 12 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 180 >> > 13 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 168 >> > 14 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 178 >> > 15 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 170 >> > 16 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 149 >> > 17 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 203 >> > 18 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 173 >> > 19 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 158 >> > 20 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 154 >> > 21 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 160 >> > 22 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 167 >> > 23 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 188 >> > 37 nvme 1.09149 1.0 1117G 1061G 57214M 95.00 90.63 98 >> > 24 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 187 >> > 25 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 200 >> > 26 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 147 >> > 27 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 171 >> > 28 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 162 >> > 29 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 152 >> > 30 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 174 >> > 31 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 176 >> > 32 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 182 >> > 33 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 155 >> > 34 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 166 >> > 35 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 176 >> > 38 nvme 1.09149 1.0 1117G 857G 260G 76.71 73.18 79 >> > TOTAL 265T 2847G 262T 1.05 >> > MIN/MAX VAR: 0.03/90.63 STDDEV: 22.81 >> > >> > 3) # ceph osd tree >> > -1 265.29291 root default >> > -388.43097 host pulpo-osd01 >> > 0 hdd 7.27829 osd.0up 1.0 1.0 >> > 1 hdd 7.27829 osd.1up 1.0 1.0 >> > 2 hdd 7.27829 osd.2up 1.0 1.0 >> > 3 hdd 7.27829 osd.3up 1.0 1.0 >> > 4 hdd 7.27829 osd.4up 1.0 1.0 >> > 5 hdd 7.27829 osd.5up 1.0 1.0 >> > 6 hdd 7.27829 osd.6up 1.0 1.0 >> > 7 hdd 7.27829 osd.7
[ceph-users] what does associating ceph pool to application do?
Hi All, Could someone tell me / link to what associating a ceph pool to an application does? I hope this info includes why "Disabling an application within a pool might result in loss of application functionality" when running 'ceph osd application disable ' Thanks! Chad. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] what does associating ceph pool to application do?
Scrolled down a bit and found this blog post: https://ceph.com/community/new-luminous-pool-tags/ If things haven't changed: Could someone tell me / link to what associating a ceph pool to an application does? ATM it's a tag and does nothing to the pool/PG/etc structure I hope this info includes why "Disabling an application within a pool might result in loss of application functionality" when running 'ceph osd application disable ' A stern warning to ignore. C. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph cache pool full
On Fri, 6 Oct 2017 09:14:40 -0700 Shawfeng Dong wrote: > I found the command: rados -p cephfs_cache cache-flush-evict-all > That's not what you want/need. Though it will fix your current "full" issue. > The documentation ( > http://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/) has > been improved a lot since I last checked it a few weeks ago! > The need to set max_bytes and max_objects has been documented for ages (since Hammer). more below... > -Shaw > > On Fri, Oct 6, 2017 at 9:10 AM, Shawfeng Dong wrote: > > > Thanks, Luis. > > > > I've just set max_bytes and max_objects: How? Editing the conf file won't help until a restart. > > target_max_objects: 100 (1M) > > target_max_bytes: 1099511627776 (1TB) > I'd lower that or the cache_target_full_ratio by another 10%. Christian > > > > but nothing appears to be happening. Is there a way to force flushing? > > > > Thanks, > > Shaw > > > > On Fri, Oct 6, 2017 at 8:55 AM, Luis Periquito > > wrote: > > > >> Not looking at anything else, you didn't set the max_bytes or > >> max_objects for it to start flushing... > >> > >> On Fri, Oct 6, 2017 at 4:49 PM, Shawfeng Dong wrote: > >> > Dear all, > >> > > >> > Thanks a lot for the very insightful comments/suggestions! > >> > > >> > There are 3 OSD servers in our pilot Ceph cluster, each with 2x 1TB SSDs > >> > (boot disks), 12x 8TB SATA HDDs and 2x 1.2TB NVMe SSDs. We use the > >> bluestore > >> > backend, with the first NVMe as the WAL and DB devices for OSDs on the > >> HDDs. > >> > And we try to create a cache tier out of the second NVMes. > >> > > >> > Here are the outputs of the commands suggested by David: > >> > > >> > 1) # ceph df > >> > GLOBAL: > >> > SIZE AVAIL RAW USED %RAW USED > >> > 265T 262T2847G 1.05 > >> > POOLS: > >> > NAMEID USED %USED MAX AVAIL > >> OBJECTS > >> > cephfs_data 1 0 0 248T > >> 0 > >> > cephfs_metadata 2 8515k 0 248T > >> 24 > >> > cephfs_cache3 1381G 100.00 0 > >> 355385 > >> > > >> > 2) # ceph osd df > >> > 0 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 174 > >> > 1 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 169 > >> > 2 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 173 > >> > 3 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 159 > >> > 4 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 173 > >> > 5 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 162 > >> > 6 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 149 > >> > 7 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 179 > >> > 8 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 163 > >> > 9 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 194 > >> > 10 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 185 > >> > 11 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 168 > >> > 36 nvme 1.09149 1.0 1117G 855G 262G 76.53 73.01 79 > >> > 12 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 180 > >> > 13 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 168 > >> > 14 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 178 > >> > 15 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 170 > >> > 16 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 149 > >> > 17 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 203 > >> > 18 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 173 > >> > 19 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 158 > >> > 20 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 154 > >> > 21 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 160 > >> > 22 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 167 > >> > 23 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 188 > >> > 37 nvme 1.09149 1.0 1117G 1061G 57214M 95.00 90.63 98 > >> > 24 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 187 > >> > 25 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 200 > >> > 26 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 147 > >> > 27 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 171 > >> > 28 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 162 > >> > 29 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 152 > >> > 30 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 174 > >> > 31 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 176 > >> > 32 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 182 > >> > 33 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 155 > >> > 34 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 166 > >> > 35 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 176 > >> > 38 nvme 1.09149 1.0 1117G 857G 260G 76.71 73.18 79 > >> > TOTAL 265T 2847G 262T 1.05 > >> > MIN/MAX VAR: 0.03/90.63 STDDEV: 22.81 > >> > > >> > 3) # ceph osd tree > >> > -1
Re: [ceph-users] Ceph cache pool full
Hi Christian, I set those via CLI: # ceph osd pool set cephfs_cache target_max_bytes 1099511627776 # ceph osd pool set cephfs_cache target_max_objects 100 but manual flushing doesn't appear to work: # rados -p cephfs_cache cache-flush-evict-all 100046a.0ca6 it just gets stuck there for a long time. Any suggestion? Do I need to restart the daemons or reboot the nodes? Thanks, Shaw On Fri, Oct 6, 2017 at 9:31 AM, Christian Balzer wrote: > On Fri, 6 Oct 2017 09:14:40 -0700 Shawfeng Dong wrote: > > > I found the command: rados -p cephfs_cache cache-flush-evict-all > > > That's not what you want/need. > Though it will fix your current "full" issue. > > > The documentation ( > > http://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/) has > > been improved a lot since I last checked it a few weeks ago! > > > The need to set max_bytes and max_objects has been documented for ages > (since Hammer). > > more below... > > > -Shaw > > > > On Fri, Oct 6, 2017 at 9:10 AM, Shawfeng Dong wrote: > > > > > Thanks, Luis. > > > > > > I've just set max_bytes and max_objects: > How? > Editing the conf file won't help until a restart. > > > > target_max_objects: 100 (1M) > > > target_max_bytes: 1099511627776 (1TB) > > > I'd lower that or the cache_target_full_ratio by another 10%. > > Christian > > > > > > but nothing appears to be happening. Is there a way to force flushing? > > > > > > Thanks, > > > Shaw > > > > > > On Fri, Oct 6, 2017 at 8:55 AM, Luis Periquito > > > wrote: > > > > > >> Not looking at anything else, you didn't set the max_bytes or > > >> max_objects for it to start flushing... > > >> > > >> On Fri, Oct 6, 2017 at 4:49 PM, Shawfeng Dong wrote: > > >> > Dear all, > > >> > > > >> > Thanks a lot for the very insightful comments/suggestions! > > >> > > > >> > There are 3 OSD servers in our pilot Ceph cluster, each with 2x 1TB > SSDs > > >> > (boot disks), 12x 8TB SATA HDDs and 2x 1.2TB NVMe SSDs. We use the > > >> bluestore > > >> > backend, with the first NVMe as the WAL and DB devices for OSDs on > the > > >> HDDs. > > >> > And we try to create a cache tier out of the second NVMes. > > >> > > > >> > Here are the outputs of the commands suggested by David: > > >> > > > >> > 1) # ceph df > > >> > GLOBAL: > > >> > SIZE AVAIL RAW USED %RAW USED > > >> > 265T 262T2847G 1.05 > > >> > POOLS: > > >> > NAMEID USED %USED MAX AVAIL > > >> OBJECTS > > >> > cephfs_data 1 0 0 248T > > >> 0 > > >> > cephfs_metadata 2 8515k 0 248T > > >> 24 > > >> > cephfs_cache3 1381G 100.00 0 > > >> 355385 > > >> > > > >> > 2) # ceph osd df > > >> > 0 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 174 > > >> > 1 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 169 > > >> > 2 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 173 > > >> > 3 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 159 > > >> > 4 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 173 > > >> > 5 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 162 > > >> > 6 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 149 > > >> > 7 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 179 > > >> > 8 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 163 > > >> > 9 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 194 > > >> > 10 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 185 > > >> > 11 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 168 > > >> > 36 nvme 1.09149 1.0 1117G 855G 262G 76.53 73.01 79 > > >> > 12 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 180 > > >> > 13 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 168 > > >> > 14 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 178 > > >> > 15 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 170 > > >> > 16 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 149 > > >> > 17 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 203 > > >> > 18 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 173 > > >> > 19 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 158 > > >> > 20 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 154 > > >> > 21 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 160 > > >> > 22 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 167 > > >> > 23 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 188 > > >> > 37 nvme 1.09149 1.0 1117G 1061G 57214M 95.00 90.63 98 > > >> > 24 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 187 > > >> > 25 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 200 > > >> > 26 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 147 > > >> > 27 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 171 > > >> > 28 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 162 > > >> > 29 hdd 7.27829 1.0 7452G 2072M 74
Re: [ceph-users] Ceph cache pool full
Notice in the URL for the documentation the use of "luminous". When you looked a few weeks ago, you might have been looking at the documentation for a different version of Ceph. You can change that to jewel, hammer, kraken, master, etc depending on which version of Ceph you are running or reading about. Google gets confused and will pull up random versions of the ceph documentation for a page. It's on us to make sure that the url is pointing to the version of Ceph that we are using. While it's sitting there in the flush command, can you see if there are any objects in the underlying data pool? Hopefully the count will be growing. On Fri, Oct 6, 2017 at 12:39 PM Shawfeng Dong wrote: > Hi Christian, > > I set those via CLI: > # ceph osd pool set cephfs_cache target_max_bytes 1099511627776 > # ceph osd pool set cephfs_cache target_max_objects 100 > > but manual flushing doesn't appear to work: > # rados -p cephfs_cache cache-flush-evict-all > 100046a.0ca6 > > it just gets stuck there for a long time. > > Any suggestion? Do I need to restart the daemons or reboot the nodes? > > Thanks, > Shaw > > > > On Fri, Oct 6, 2017 at 9:31 AM, Christian Balzer wrote: > >> On Fri, 6 Oct 2017 09:14:40 -0700 Shawfeng Dong wrote: >> >> > I found the command: rados -p cephfs_cache cache-flush-evict-all >> > >> That's not what you want/need. >> Though it will fix your current "full" issue. >> >> > The documentation ( >> > http://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/) has >> > been improved a lot since I last checked it a few weeks ago! >> > >> The need to set max_bytes and max_objects has been documented for ages >> (since Hammer). >> >> more below... >> >> > -Shaw >> > >> > On Fri, Oct 6, 2017 at 9:10 AM, Shawfeng Dong wrote: >> > >> > > Thanks, Luis. >> > > >> > > I've just set max_bytes and max_objects: >> How? >> Editing the conf file won't help until a restart. >> >> > > target_max_objects: 100 (1M) >> > > target_max_bytes: 1099511627776 (1TB) >> > >> I'd lower that or the cache_target_full_ratio by another 10%. >> >> Christian >> > > >> > > but nothing appears to be happening. Is there a way to force flushing? >> > > >> > > Thanks, >> > > Shaw >> > > >> > > On Fri, Oct 6, 2017 at 8:55 AM, Luis Periquito >> > > wrote: >> > > >> > >> Not looking at anything else, you didn't set the max_bytes or >> > >> max_objects for it to start flushing... >> > >> >> > >> On Fri, Oct 6, 2017 at 4:49 PM, Shawfeng Dong wrote: >> > >> > Dear all, >> > >> > >> > >> > Thanks a lot for the very insightful comments/suggestions! >> > >> > >> > >> > There are 3 OSD servers in our pilot Ceph cluster, each with 2x >> 1TB SSDs >> > >> > (boot disks), 12x 8TB SATA HDDs and 2x 1.2TB NVMe SSDs. We use the >> > >> bluestore >> > >> > backend, with the first NVMe as the WAL and DB devices for OSDs on >> the >> > >> HDDs. >> > >> > And we try to create a cache tier out of the second NVMes. >> > >> > >> > >> > Here are the outputs of the commands suggested by David: >> > >> > >> > >> > 1) # ceph df >> > >> > GLOBAL: >> > >> > SIZE AVAIL RAW USED %RAW USED >> > >> > 265T 262T2847G 1.05 >> > >> > POOLS: >> > >> > NAMEID USED %USED MAX AVAIL >> > >> OBJECTS >> > >> > cephfs_data 1 0 0 248T >> > >> 0 >> > >> > cephfs_metadata 2 8515k 0 248T >> > >> 24 >> > >> > cephfs_cache3 1381G 100.00 0 >> > >> 355385 >> > >> > >> > >> > 2) # ceph osd df >> > >> > 0 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 174 >> > >> > 1 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 169 >> > >> > 2 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 173 >> > >> > 3 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 159 >> > >> > 4 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 173 >> > >> > 5 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 162 >> > >> > 6 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 149 >> > >> > 7 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 179 >> > >> > 8 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 163 >> > >> > 9 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 194 >> > >> > 10 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 185 >> > >> > 11 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 168 >> > >> > 36 nvme 1.09149 1.0 1117G 855G 262G 76.53 73.01 79 >> > >> > 12 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 180 >> > >> > 13 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 168 >> > >> > 14 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 178 >> > >> > 15 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 170 >> > >> > 16 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 149 >> > >> > 17 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 203 >> > >> > 18 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.
Re: [ceph-users] what does associating ceph pool to application do?
On Fri, Oct 6, 2017 at 5:28 PM, Chad William Seys wrote: > Scrolled down a bit and found this blog post: > https://ceph.com/community/new-luminous-pool-tags/ > > If things haven't changed: > >>Could someone tell me / link to what associating a ceph pool to an >> application does? > > > ATM it's a tag and does nothing to the pool/PG/etc structure > >>I hope this info includes why "Disabling an application within a pool >> might result in loss of application functionality" when running 'ceph osd >> application disable ' > > > A stern warning to ignore. Just to stern this up a bit... In the future, you may find that things stop working if you remove the application tags. For example, in Mimic, application tags will be involved in authentication for CephFS, and if you removed the cephfs tags from your data pool, your clients would stop being able to access it. Cheers, John > > C. > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph cache pool full
Curiously, it has been quite a while, but there is still no object in the underlying data pool: # rados -p cephfs_data ls Any advice? On Fri, Oct 6, 2017 at 9:45 AM, David Turner wrote: > Notice in the URL for the documentation the use of "luminous". When you > looked a few weeks ago, you might have been looking at the documentation > for a different version of Ceph. You can change that to jewel, hammer, > kraken, master, etc depending on which version of Ceph you are running or > reading about. Google gets confused and will pull up random versions of > the ceph documentation for a page. It's on us to make sure that the url is > pointing to the version of Ceph that we are using. > > While it's sitting there in the flush command, can you see if there are > any objects in the underlying data pool? Hopefully the count will be > growing. > > On Fri, Oct 6, 2017 at 12:39 PM Shawfeng Dong wrote: > >> Hi Christian, >> >> I set those via CLI: >> # ceph osd pool set cephfs_cache target_max_bytes 1099511627776 >> # ceph osd pool set cephfs_cache target_max_objects 100 >> >> but manual flushing doesn't appear to work: >> # rados -p cephfs_cache cache-flush-evict-all >> 100046a.0ca6 >> >> it just gets stuck there for a long time. >> >> Any suggestion? Do I need to restart the daemons or reboot the nodes? >> >> Thanks, >> Shaw >> >> >> >> On Fri, Oct 6, 2017 at 9:31 AM, Christian Balzer wrote: >> >>> On Fri, 6 Oct 2017 09:14:40 -0700 Shawfeng Dong wrote: >>> >>> > I found the command: rados -p cephfs_cache cache-flush-evict-all >>> > >>> That's not what you want/need. >>> Though it will fix your current "full" issue. >>> >>> > The documentation ( >>> > http://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/) >>> has >>> > been improved a lot since I last checked it a few weeks ago! >>> > >>> The need to set max_bytes and max_objects has been documented for ages >>> (since Hammer). >>> >>> more below... >>> >>> > -Shaw >>> > >>> > On Fri, Oct 6, 2017 at 9:10 AM, Shawfeng Dong wrote: >>> > >>> > > Thanks, Luis. >>> > > >>> > > I've just set max_bytes and max_objects: >>> How? >>> Editing the conf file won't help until a restart. >>> >>> > > target_max_objects: 100 (1M) >>> > > target_max_bytes: 1099511627776 (1TB) >>> > >>> I'd lower that or the cache_target_full_ratio by another 10%. >>> >>> Christian >>> > > >>> > > but nothing appears to be happening. Is there a way to force >>> flushing? >>> > > >>> > > Thanks, >>> > > Shaw >>> > > >>> > > On Fri, Oct 6, 2017 at 8:55 AM, Luis Periquito >>> > > wrote: >>> > > >>> > >> Not looking at anything else, you didn't set the max_bytes or >>> > >> max_objects for it to start flushing... >>> > >> >>> > >> On Fri, Oct 6, 2017 at 4:49 PM, Shawfeng Dong >>> wrote: >>> > >> > Dear all, >>> > >> > >>> > >> > Thanks a lot for the very insightful comments/suggestions! >>> > >> > >>> > >> > There are 3 OSD servers in our pilot Ceph cluster, each with 2x >>> 1TB SSDs >>> > >> > (boot disks), 12x 8TB SATA HDDs and 2x 1.2TB NVMe SSDs. We use the >>> > >> bluestore >>> > >> > backend, with the first NVMe as the WAL and DB devices for OSDs >>> on the >>> > >> HDDs. >>> > >> > And we try to create a cache tier out of the second NVMes. >>> > >> > >>> > >> > Here are the outputs of the commands suggested by David: >>> > >> > >>> > >> > 1) # ceph df >>> > >> > GLOBAL: >>> > >> > SIZE AVAIL RAW USED %RAW USED >>> > >> > 265T 262T2847G 1.05 >>> > >> > POOLS: >>> > >> > NAMEID USED %USED MAX AVAIL >>> > >> OBJECTS >>> > >> > cephfs_data 1 0 0 248T >>> > >> 0 >>> > >> > cephfs_metadata 2 8515k 0 248T >>> > >> 24 >>> > >> > cephfs_cache3 1381G 100.00 0 >>> > >> 355385 >>> > >> > >>> > >> > 2) # ceph osd df >>> > >> > 0 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 174 >>> > >> > 1 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 169 >>> > >> > 2 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 173 >>> > >> > 3 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 159 >>> > >> > 4 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 173 >>> > >> > 5 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 162 >>> > >> > 6 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 149 >>> > >> > 7 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 179 >>> > >> > 8 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 163 >>> > >> > 9 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 194 >>> > >> > 10 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 185 >>> > >> > 11 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 168 >>> > >> > 36 nvme 1.09149 1.0 1117G 855G 262G 76.53 73.01 79 >>> > >> > 12 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 180 >>> > >> > 13 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 168 >>> > >> >
Re: [ceph-users] "ceph osd status" fails
On Fri, Oct 6, 2017 at 5:01 PM, Richard Hesketh wrote: > When I try to run the command "ceph osd status" on my cluster, I just get an > error. Luckily unlike the last issue I had with ceph fs commands it doesn't > seem to be crashing any of the daemons. > > root@vm-ds-01:/var/log/ceph# ceph osd status > Error EINVAL: Traceback (most recent call last): > File "/usr/lib/ceph/mgr/status/module.py", line 293, in handle_command > return self.handle_osd_status(cmd) > File "/usr/lib/ceph/mgr/status/module.py", line 273, in handle_osd_status > stats = osd_stats[osd_id] > KeyError: (78L,) Looks like this will happen if an OSD is in the OSDMap but for whatever reason isn't present in the statistics stored in the PG map. Possibly the OSD has no PGs, or something is wrong with how the OSDs report PGs to the manager. Ticket here: http://tracker.ceph.com/issues/21707. If you are proficient in python you can quickly add an exception handler to the code and things will be okay. The other (crashing) backtrace in your log is http://tracker.ceph.com/issues/17737, which is pending backport of the fix. Cheers, John > > Example and relevant excerpt from the ceph-mgr log shown at > https://gist.github.com/rjhesketh/378ec118e42289a2dd0b1dd2462aae92 > > Is this trying to poll stats for an OSD which doesn't exist and therefore > breaking? > > Rich > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] ceph-volume: migration and disk partition support
Hi, Now that ceph-volume is part of the Luminous release, we've been able to provide filestore support for LVM-based OSDs. We are making use of LVM's powerful mechanisms to store metadata which allows the process to no longer rely on UDEV and GPT labels (unlike ceph-disk). Bluestore support should be the next step for `ceph-volume lvm`, and while that is planned we are thinking of ways to improve the current caveats (like OSDs not coming up) for clusters that have deployed OSDs with ceph-disk. --- New clusters --- The `ceph-volume lvm` deployment is straightforward (currently supported in ceph-ansible), but there isn't support for plain disks (with partitions) currently, like there is with ceph-disk. Is there a pressing interest in supporting plain disks with partitions? Or only supporting LVM-based OSDs fine? --- Existing clusters --- Migration to ceph-volume, even with plain disk support means re-creating the OSD from scratch, which would end up moving data. There is no way to make a GPT/ceph-disk OSD become a ceph-volume one without starting from scratch. A temporary workaround would be to provide a way for existing OSDs to be brought up without UDEV and ceph-disk, by creating logic in ceph-volume that could load them with systemd directly. This wouldn't make them lvm-based, nor it would mean there is direct support for them, just a temporary workaround to make them start without UDEV and ceph-disk. I'm interested in what current users might look for here,: is it fine to provide this workaround if the issues are that problematic? Or is it OK to plan a migration towards ceph-volume OSDs? -Alfredo ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] v10.2.10 Jewel released
This point release brings a number of important bugfixes in all major components of Ceph, we recommend all Jewel 10.2.x users to upgrade. For more details checkout the release notes entry at the ceph blog http://ceph.com/releases/v10-2-10-jewel-released/ Notable Changes - * build/ops: Add fix subcommand to ceph-disk, fix SELinux denials, and speed up upgrade from non-SELinux enabled ceph to an SELinux enabled one (issue#20077, issue#20184, issue#19545, pr#14346, Boris Ranto) * build/ops: deb: Fix logrotate packaging (issue#19938, pr#15428, Nathan Cutler) * build/ops: extended, customizable systemd ceph-disk timeout (issue#18740, pr#15051, Alexey Sheplyakov) * build/ops: rpm: fix python-Sphinx package name for SUSE (issue#19924, pr#15196, Nathan Cutler, Jan Matejek) * build/ops: rpm: set subman cron attributes in spec file (issue#20074, pr#15473, Thomas Serlin) * cephfs: ceph-fuse segfaults at mount time, assert in ceph::log::Log::stop (issue#18157, pr#16963, Greg Farnum) * cephfs: df reports negative disk "used" value when quota exceed (issue#20178, pr#16151, John Spray) * cephfs: get_quota_root sends lookupname op for every buffered write (issue#20945, pr#17396, Dan van der Ster) * cephfs: osdc/Filer: truncate large file party by party (issue#19755, pr#15442, "Yan, Zheng") * core: an OSD was seen getting ENOSPC even with osd_failsafe_full_ratio passed (issue#20544, issue#16878, issue#19733, issue#15912, pr#15050, Sage Weil, David Zafman) * core: disable skewed utilization warning by default (issue#20730, pr#17210, David Zafman) * core: interval_set: optimize intersect_of insert operations (issue#21229, pr#17514, Zac Medico) * core: kv: let ceph_logger destructed after db reset (issue#21336, pr#17626, wumingqiao) * core: test_envlibrados_for_rocksdb.yaml fails on crypto restart (issue#19741, pr#16293, Kefu Chai) * libradosstriper silently fails to delete empty objects in jewel (issue#20325, pr#15760, Stan K) * librbd: fail IO request when exclusive lock cannot be obtained (issue#20168, issue#21251, pr#17402, Jason Dillaman) * librbd: prevent self-blacklisting during break lock (issue#18666, pr#17412, Jason Dillaman) * librbd: reacquire lock should update lock owner client id (issue#19929, pr#17385, Jason Dillaman) * mds: damage reporting by ino number is useless (issue#18509, issue#16016, pr#14699, John Spray, Michal Jarzabek) * mds: log rotation doesn't work if mds has respawned (issue#19291, pr#14673, Patrick Donnelly) * mds: save projected path into inode_t::stray_prior_path (issue#20340, pr#16150, "Yan, Zheng") * mon: crash on shutdown, lease_ack_timeout event (issue#19825, pr#15083, Kefu Chai, Michal Jarzabek, Alexey Sheplyakov) * mon: Disallow enabling 'hashpspool' option to a pool without some kind of --i-understand-this-will-remap-all-pgs flag (issue#18468, pr#13507, Vikhyat Umrao) * mon: factor mon_osd_full_ratio into MAX AVAIL calc (issue#18522, pr#15236, Sage Weil) * mon: fail to form large quorum; msg/async busy loop (issue#20230, pr#15726, Haomai Wang, Michal Jarzabek) * mon: fix force_pg_create pg stuck in creating bug (issue#18298, pr#17008, Alexey Sheplyakov) * mon: osd crush set crushmap need sanity check (issue#19302, pr#16144, Loic Dachary) * osd: Add heartbeat message for Jumbo Frames (MTU 9000) (issue#20087, issue#20323, pr#16059, Piotr Dałek, Sage Weil, Greg Farnum) * osd: fix infinite loops in fiemap (issue#19996, pr#15189, Sage Weil, Ning Yao) * osd: leaked MOSDMap (issue#18293, pr#14943, Sage Weil) * osd: objecter full_try behavior not consistent with osd (issue#19430, pr#15474, Sage Weil) * osd: omap threadpool heartbeat is only reset every 100 values (issue#20375, pr#16167, Josh Durgin) * osd: osd_internal_types: wake snaptrimmer on put_read lock, too (issue#19131, pr#16015, Sage Weil) * osd: PrimaryLogPG: do not call on_shutdown() if (pg.deleting) (issue#19902, pr#15065, Kefu Chai) * osd: rados ls on pool with no access returns no error (issue#20043, issue#19790, pr#16473, Nathan Cutler, Kefu Chai, John Spray, Sage Weil, Brad Hubbard) * osd: ReplicatedPG: solve cache tier osd high memory consumption (issue#20464, pr#16169, Peng Xie) * osd: Reset() snaptrimmer on shutdown and do not default-abort on leaked pg refs (issue#19931, pr#15322, Greg Farnum) * osd: scrub_to specifies clone ver, but transaction include head write ver (issue#20041, pr#16405, David Zafman) * osd: unlock sdata_op_ordering_lock with sdata_lock hold to avoid missing wakeup signal (issue#20427, pr#15947, Alexey Sheplyakov) * qa: add a sleep after restarting osd before "tell"ing it (issue#16239, pr#15475, Kefu Chai) * rbd: api: is_exclusive_lock_owner shouldn't return -EBUSY (issue#20182, pr#16296, Jason Dillaman) * rbd: cli: ensure positional arguments exist before casting (issue#20185, pr#16295, Jason Dillaman) * rbd: cli: map with cephx disabled results in error message (issue#19035, pr#16297, Jason Dillaman) * rbd: default features should be nego
Re: [ceph-users] what does associating ceph pool to application do?
Thanks John! I see that a pool can have more than one "application". Should I feel free to combine uses (e.g. cephfs,rbd) or is this counterindicated? Thanks! Chad. Just to stern this up a bit... In the future, you may find that things stop working if you remove the application tags. For example, in Mimic, application tags will be involved in authentication for CephFS, and if you removed the cephfs tags from your data pool, your clients would stop being able to access it. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph cache pool full
All of this data is test data, yeah? I would start by removing the cache-tier and pool, recreate it and attach it, configure all of the settings including the maximums, and start testing things again. I would avoid doing the 1.3TB file test until after you've confirmed that the smaller files are being flushed appropriately to the data pool (manually flushing/evicting it) and then scale up your testing to the larger files. On Fri, Oct 6, 2017 at 12:54 PM Shawfeng Dong wrote: > Curiously, it has been quite a while, but there is still no object in the > underlying data pool: > # rados -p cephfs_data ls > > Any advice? > > On Fri, Oct 6, 2017 at 9:45 AM, David Turner > wrote: > >> Notice in the URL for the documentation the use of "luminous". When you >> looked a few weeks ago, you might have been looking at the documentation >> for a different version of Ceph. You can change that to jewel, hammer, >> kraken, master, etc depending on which version of Ceph you are running or >> reading about. Google gets confused and will pull up random versions of >> the ceph documentation for a page. It's on us to make sure that the url is >> pointing to the version of Ceph that we are using. >> >> While it's sitting there in the flush command, can you see if there are >> any objects in the underlying data pool? Hopefully the count will be >> growing. >> >> On Fri, Oct 6, 2017 at 12:39 PM Shawfeng Dong wrote: >> >>> Hi Christian, >>> >>> I set those via CLI: >>> # ceph osd pool set cephfs_cache target_max_bytes 1099511627776 >>> # ceph osd pool set cephfs_cache target_max_objects 100 >>> >>> but manual flushing doesn't appear to work: >>> # rados -p cephfs_cache cache-flush-evict-all >>> 100046a.0ca6 >>> >>> it just gets stuck there for a long time. >>> >>> Any suggestion? Do I need to restart the daemons or reboot the nodes? >>> >>> Thanks, >>> Shaw >>> >>> >>> >>> On Fri, Oct 6, 2017 at 9:31 AM, Christian Balzer wrote: >>> On Fri, 6 Oct 2017 09:14:40 -0700 Shawfeng Dong wrote: > I found the command: rados -p cephfs_cache cache-flush-evict-all > That's not what you want/need. Though it will fix your current "full" issue. > The documentation ( > http://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/) has > been improved a lot since I last checked it a few weeks ago! > The need to set max_bytes and max_objects has been documented for ages (since Hammer). more below... > -Shaw > > On Fri, Oct 6, 2017 at 9:10 AM, Shawfeng Dong wrote: > > > Thanks, Luis. > > > > I've just set max_bytes and max_objects: How? Editing the conf file won't help until a restart. > > target_max_objects: 100 (1M) > > target_max_bytes: 1099511627776 (1TB) > I'd lower that or the cache_target_full_ratio by another 10%. Christian > > > > but nothing appears to be happening. Is there a way to force flushing? > > > > Thanks, > > Shaw > > > > On Fri, Oct 6, 2017 at 8:55 AM, Luis Periquito >>> > > > wrote: > > > >> Not looking at anything else, you didn't set the max_bytes or > >> max_objects for it to start flushing... > >> > >> On Fri, Oct 6, 2017 at 4:49 PM, Shawfeng Dong wrote: > >> > Dear all, > >> > > >> > Thanks a lot for the very insightful comments/suggestions! > >> > > >> > There are 3 OSD servers in our pilot Ceph cluster, each with 2x 1TB SSDs > >> > (boot disks), 12x 8TB SATA HDDs and 2x 1.2TB NVMe SSDs. We use the > >> bluestore > >> > backend, with the first NVMe as the WAL and DB devices for OSDs on the > >> HDDs. > >> > And we try to create a cache tier out of the second NVMes. > >> > > >> > Here are the outputs of the commands suggested by David: > >> > > >> > 1) # ceph df > >> > GLOBAL: > >> > SIZE AVAIL RAW USED %RAW USED > >> > 265T 262T2847G 1.05 > >> > POOLS: > >> > NAMEID USED %USED MAX AVAIL > >> OBJECTS > >> > cephfs_data 1 0 0 248T > >> 0 > >> > cephfs_metadata 2 8515k 0 248T > >> 24 > >> > cephfs_cache3 1381G 100.00 0 > >> 355385 > >> > > >> > 2) # ceph osd df > >> > 0 hdd 7.27829 1.0 7452G 2076M 7450G 0.03 0.03 174 > >> > 1 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 169 > >> > 2 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 173 > >> > 3 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 159 > >> > 4 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 173 > >> > 5 hdd 7.27829 1.0 7452G 2072M 7450G 0.03 0.03 162 > >> > 6 hdd 7.27829 1
Re: [ceph-users] what does associating ceph pool to application do?
Hello Chad, On Fri, Oct 6, 2017 at 10:01 AM, Chad William Seys wrote: > Thanks John! I see that a pool can have more than one "application". Should > I feel free to combine uses (e.g. cephfs,rbd) or is this counterindicated? That's not currently possible but we are thinking about changes which would allow multiple ceph file systems to use the same data pool by having each FS work in a separate namespace. See also: http://tracker.ceph.com/issues/15066 Support with CephFS and RBD using the same pool may follow that. -- Patrick Donnelly ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph cache pool full
Here is a quick update. I found that a CephFS client process was accessing the big 1TB file, which I think had a lock on the file, preventing the flushing of objects to the underlying data pool. Once I killed that process, objects started to flush to the data pool automatically (with target_max_bytes & target_max_objects set); and I can force the flushing with 'rados -p cephfs_cache cache-flush-evict-all' as well. So David appears to be right in saying that "it can only hold full files and not flush partial files". This will be problematic if we want to transfer a file that is bigger in size than the cache pool! We did this whole scheme (EC data pool plus NVMe cache tier) just for experimentation. I've learned a lot from the experiment and from your guys. Thank you very much! For production, I think I'll simply use a replicated pool for data on the HDDs (with bluestore WAL and DB on the 1st NVMe), and a replicated pool for metadata on the 2nd NVMe. Please let me know if you have any further advice / suggestion. Best, Shaw On Fri, Oct 6, 2017 at 10:07 AM, David Turner wrote: > All of this data is test data, yeah? I would start by removing the > cache-tier and pool, recreate it and attach it, configure all of the > settings including the maximums, and start testing things again. I would > avoid doing the 1.3TB file test until after you've confirmed that the > smaller files are being flushed appropriately to the data pool (manually > flushing/evicting it) and then scale up your testing to the larger files. > On Fri, Oct 6, 2017 at 12:54 PM Shawfeng Dong wrote: > >> Curiously, it has been quite a while, but there is still no object in the >> underlying data pool: >> # rados -p cephfs_data ls >> >> Any advice? >> >> On Fri, Oct 6, 2017 at 9:45 AM, David Turner >> wrote: >> >>> Notice in the URL for the documentation the use of "luminous". When you >>> looked a few weeks ago, you might have been looking at the documentation >>> for a different version of Ceph. You can change that to jewel, hammer, >>> kraken, master, etc depending on which version of Ceph you are running or >>> reading about. Google gets confused and will pull up random versions of >>> the ceph documentation for a page. It's on us to make sure that the url is >>> pointing to the version of Ceph that we are using. >>> >>> While it's sitting there in the flush command, can you see if there are >>> any objects in the underlying data pool? Hopefully the count will be >>> growing. >>> >>> On Fri, Oct 6, 2017 at 12:39 PM Shawfeng Dong wrote: >>> Hi Christian, I set those via CLI: # ceph osd pool set cephfs_cache target_max_bytes 1099511627776 # ceph osd pool set cephfs_cache target_max_objects 100 but manual flushing doesn't appear to work: # rados -p cephfs_cache cache-flush-evict-all 100046a.0ca6 it just gets stuck there for a long time. Any suggestion? Do I need to restart the daemons or reboot the nodes? Thanks, Shaw On Fri, Oct 6, 2017 at 9:31 AM, Christian Balzer wrote: > On Fri, 6 Oct 2017 09:14:40 -0700 Shawfeng Dong wrote: > > > I found the command: rados -p cephfs_cache cache-flush-evict-all > > > That's not what you want/need. > Though it will fix your current "full" issue. > > > The documentation ( > > http://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/) > has > > been improved a lot since I last checked it a few weeks ago! > > > The need to set max_bytes and max_objects has been documented for ages > (since Hammer). > > more below... > > > -Shaw > > > > On Fri, Oct 6, 2017 at 9:10 AM, Shawfeng Dong wrote: > > > > > Thanks, Luis. > > > > > > I've just set max_bytes and max_objects: > How? > Editing the conf file won't help until a restart. > > > > target_max_objects: 100 (1M) > > > target_max_bytes: 1099511627776 (1TB) > > > I'd lower that or the cache_target_full_ratio by another 10%. > > Christian > > > > > > but nothing appears to be happening. Is there a way to force > flushing? > > > > > > Thanks, > > > Shaw > > > > > > On Fri, Oct 6, 2017 at 8:55 AM, Luis Periquito < > periqu...@gmail.com> > > > wrote: > > > > > >> Not looking at anything else, you didn't set the max_bytes or > > >> max_objects for it to start flushing... > > >> > > >> On Fri, Oct 6, 2017 at 4:49 PM, Shawfeng Dong > wrote: > > >> > Dear all, > > >> > > > >> > Thanks a lot for the very insightful comments/suggestions! > > >> > > > >> > There are 3 OSD servers in our pilot Ceph cluster, each with 2x > 1TB SSDs > > >> > (boot disks), 12x 8TB SATA HDDs and 2x 1.2TB NVMe SSDs. We use > the > > >> bluestore > > >> > backend, with the first NVMe as the WAL and DB devices for OSDs
Re: [ceph-users] Ceph cache pool full
You can still use EC for CephFS without a cache tier since you are using Luminous. This is new functionality since Luminous was released while the majority of guides you will see are for setups on Jewel and older versions of ceph. Here's the docs regarding this including how to do it. http://docs.ceph.com/docs/luminous/rados/operations/erasure-code/#erasure-coding-with-overwrites On Fri, Oct 6, 2017, 5:22 PM Shawfeng Dong wrote: > Here is a quick update. I found that a CephFS client process was accessing > the big 1TB file, which I think had a lock on the file, preventing the > flushing of objects to the underlying data pool. Once I killed that > process, objects started to flush to the data pool automatically (with > target_max_bytes & target_max_objects set); and I can force the flushing > with 'rados -p cephfs_cache cache-flush-evict-all' as well. So David > appears to be right in saying that "it can only hold full files and not > flush partial files". This will be problematic if we want to transfer a > file that is bigger in size than the cache pool! > > We did this whole scheme (EC data pool plus NVMe cache tier) just for > experimentation. I've learned a lot from the experiment and from your guys. > Thank you very much! > > For production, I think I'll simply use a replicated pool for data on the > HDDs (with bluestore WAL and DB on the 1st NVMe), and a replicated pool for > metadata on the 2nd NVMe. Please let me know if you have any further > advice / suggestion. > > Best, > Shaw > > > > On Fri, Oct 6, 2017 at 10:07 AM, David Turner > wrote: > >> All of this data is test data, yeah? I would start by removing the >> cache-tier and pool, recreate it and attach it, configure all of the >> settings including the maximums, and start testing things again. I would >> avoid doing the 1.3TB file test until after you've confirmed that the >> smaller files are being flushed appropriately to the data pool (manually >> flushing/evicting it) and then scale up your testing to the larger files. >> On Fri, Oct 6, 2017 at 12:54 PM Shawfeng Dong wrote: >> >>> Curiously, it has been quite a while, but there is still no object in >>> the underlying data pool: >>> # rados -p cephfs_data ls >>> >>> Any advice? >>> >>> On Fri, Oct 6, 2017 at 9:45 AM, David Turner >>> wrote: >>> Notice in the URL for the documentation the use of "luminous". When you looked a few weeks ago, you might have been looking at the documentation for a different version of Ceph. You can change that to jewel, hammer, kraken, master, etc depending on which version of Ceph you are running or reading about. Google gets confused and will pull up random versions of the ceph documentation for a page. It's on us to make sure that the url is pointing to the version of Ceph that we are using. While it's sitting there in the flush command, can you see if there are any objects in the underlying data pool? Hopefully the count will be growing. On Fri, Oct 6, 2017 at 12:39 PM Shawfeng Dong wrote: > Hi Christian, > > I set those via CLI: > # ceph osd pool set cephfs_cache target_max_bytes 1099511627776 > # ceph osd pool set cephfs_cache target_max_objects 100 > > but manual flushing doesn't appear to work: > # rados -p cephfs_cache cache-flush-evict-all > 100046a.0ca6 > > it just gets stuck there for a long time. > > Any suggestion? Do I need to restart the daemons or reboot the nodes? > > Thanks, > Shaw > > > > On Fri, Oct 6, 2017 at 9:31 AM, Christian Balzer > wrote: > >> On Fri, 6 Oct 2017 09:14:40 -0700 Shawfeng Dong wrote: >> >> > I found the command: rados -p cephfs_cache cache-flush-evict-all >> > >> That's not what you want/need. >> Though it will fix your current "full" issue. >> >> > The documentation ( >> > http://docs.ceph.com/docs/luminous/rados/operations/cache-tiering/) >> has >> > been improved a lot since I last checked it a few weeks ago! >> > >> The need to set max_bytes and max_objects has been documented for ages >> (since Hammer). >> >> more below... >> >> > -Shaw >> > >> > On Fri, Oct 6, 2017 at 9:10 AM, Shawfeng Dong >> wrote: >> > >> > > Thanks, Luis. >> > > >> > > I've just set max_bytes and max_objects: >> How? >> Editing the conf file won't help until a restart. >> >> > > target_max_objects: 100 (1M) >> > > target_max_bytes: 1099511627776 (1TB) >> > >> I'd lower that or the cache_target_full_ratio by another 10%. >> >> Christian >> > > >> > > but nothing appears to be happening. Is there a way to force >> flushing? >> > > >> > > Thanks, >> > > Shaw >> > > >> > > On Fri, Oct 6, 2017 at 8:55 AM, Luis Periquito < >> periqu...@gmail.com> >> > > wrote: >> > > >> > >
