I am also seeing BlueFS spill since updating to Nautilus. I also see high
slow_used_bytes and slow_total_bytes metrics. It sure looks to me that the
only solution is to zap and rebuilt the osd. I had to manually check 36
osds some of them traditional processes and some containerized. The lack
of tooling here is underwhelming... As soon as I rebuilt the osd the
"BlueFS spill..." warning went away.
I use 50Gb db partitions on an nvme with 3 or 6 Tb spinning disks. I don't
understand the spillover.
On Fri, Feb 15, 2019 at 12:33 PM David Turner <drakonst...@gmail.com> wrote:
> The answer is probably going to be in how big your DB partition is vs how
> big your HDD disk is. From your output it looks like you have a 6TB HDD
> with a 28GB Blocks.DB partition. Even though the DB used size isn't
> currently full, I would guess that at some point since this OSD was created
> that it did fill up and what you're seeing is the part of the DB that
> spilled over to the data disk. This is why the official recommendation
> (that is quite cautious, but cautious because some use cases will use this
> up) for a blocks.db partition is 4% of the data drive. For your 6TB disks
> that's a recommendation of 240GB per DB partition. Of course the actual
> size of the DB needed is dependent on your use case. But pretty much every
> use case for a 6TB disk needs a bigger partition than 28GB.
>
> On Thu, Feb 14, 2019 at 11:58 PM Konstantin Shalygin <k0...@k0ste.ru>
> wrote:
>
>> Wrong metadata paste of osd.73 in previous message.
>>
>>
>> {
>>
>> "id": 73,
>> "arch": "x86_64",
>> "back_addr": "10.10.10.6:6804/175338",
>> "back_iface": "vlan3",
>> "bluefs": "1",
>> "bluefs_db_access_mode": "blk",
>> "bluefs_db_block_size": "4096",
>> "bluefs_db_dev": "259:22",
>> "bluefs_db_dev_node": "nvme2n1",
>> "bluefs_db_driver": "KernelDevice",
>> "bluefs_db_model": "INTEL SSDPEDMD400G4 ",
>> "bluefs_db_partition_path": "/dev/nvme2n1p11",
>> "bluefs_db_rotational": "0",
>> "bluefs_db_serial": "CVFT4324002Q400BGN ",
>> "bluefs_db_size": "30064771072",
>> "bluefs_db_type": "nvme",
>> "bluefs_single_shared_device": "0",
>> "bluefs_slow_access_mode": "blk",
>> "bluefs_slow_block_size": "4096",
>> "bluefs_slow_dev": "8:176",
>> "bluefs_slow_dev_node": "sdl",
>> "bluefs_slow_driver": "KernelDevice",
>> "bluefs_slow_model": "TOSHIBA HDWE160 ",
>> "bluefs_slow_partition_path": "/dev/sdl2",
>> "bluefs_slow_rotational": "1",
>> "bluefs_slow_size": "6001069199360",
>> "bluefs_slow_type": "hdd",
>> "bluefs_wal_access_mode": "blk",
>> "bluefs_wal_block_size": "4096",
>> "bluefs_wal_dev": "259:22",
>> "bluefs_wal_dev_node": "nvme2n1",
>> "bluefs_wal_driver": "KernelDevice",
>> "bluefs_wal_model": "INTEL SSDPEDMD400G4 ",
>> "bluefs_wal_partition_path": "/dev/nvme2n1p12",
>> "bluefs_wal_rotational": "0",
>> "bluefs_wal_serial": "CVFT4324002Q400BGN ",
>> "bluefs_wal_size": "1073741824",
>> "bluefs_wal_type": "nvme",
>> "bluestore_bdev_access_mode": "blk",
>> "bluestore_bdev_block_size": "4096",
>> "bluestore_bdev_dev": "8:176",
>> "bluestore_bdev_dev_node": "sdl",
>> "bluestore_bdev_driver": "KernelDevice",
>> "bluestore_bdev_model": "TOSHIBA HDWE160 ",
>> "bluestore_bdev_partition_path": "/dev/sdl2",
>> "bluestore_bdev_rotational": "1",
>> "bluestore_bdev_size": "6001069199360",
>> "bluestore_bdev_type": "hdd",
>> "ceph_version": "ceph version 12.2.10
>> (177915764b752804194937482a39e95e0ca3de94) luminous (stable)",
>> "cpu": "Intel(R) Xeon(R) CPU E5-2609 v4 @ 1.70GHz",
>> "default_device_class": "hdd",
>> "distro": "centos",
>> "distro_description": "CentOS Linux 7 (Core)",
>> "distro_version": "7",
>> "front_addr": "172.16.16.16:6803/175338",
>> "front_iface": "vlan4",
>> "hb_back_addr": "10.10.10.6:6805/175338",
>> "hb_front_addr": "172.16.16.16:6805/175338",
>> "hostname": "ceph-osd5",
>> "journal_rotational": "0",
>> "kernel_description": "#1 SMP Tue Aug 14 21:49:04 UTC 2018",
>> "kernel_version": "3.10.0-862.11.6.el7.x86_64",
>> "mem_swap_kb": "0",
>> "mem_total_kb": "65724256",
>> "os": "Linux",
>> "osd_data": "/var/lib/ceph/osd/ceph-73",
>> "osd_objectstore": "bluestore",
>> "rotational": "1"
>> }
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
