Do you use size 1 for your data? You know that's bad, right? Please
share 'ceph osd pool ls detail'.
Also, it's recommended to use a power of 2 for pg_num, so you should
decrease the pg_num for the pool cephfs.cephfs.data.
Zitat von Mihai Ciubancan <mihai.ciuban...@eli-np.ro>:
Hi Anthony,
Thanks for the answer:
The output of 'ceph osd df' is:
ceph osd df tree
ID CLASS WEIGHT REWEIGHT SIZE RAW USE DATA OMAP
META AVAIL %USE VAR PGS STATUS TYPE NAME
-1 167.64825 - 168 TiB 12 TiB 12 TiB 100 MiB
27 GiB 156 TiB 7.13 1.00 - root default
-3 167.64825 - 168 TiB 12 TiB 12 TiB 100 MiB
27 GiB 156 TiB 7.13 1.00 - host sto-core-hpc01
0 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB
28 MiB 14 TiB 0 0 0 up osd.0
1 ssd 13.97069 1.00000 14 TiB 12 TiB 12 TiB 6 KiB
26 GiB 2.0 TiB 85.53 12.00 129 up osd.1
2 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB
28 MiB 14 TiB 0 0 0 up osd.2
3 ssd 13.97069 1.00000 14 TiB 1.7 GiB 258 MiB 100 MiB
1.3 GiB 14 TiB 0.01 0.00 16 up osd.3
4 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB
28 MiB 14 TiB 0 0 0 up osd.4
5 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB
28 MiB 14 TiB 0 0 0 up osd.5
6 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB
28 MiB 14 TiB 0 0 0 up osd.6
7 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB
28 MiB 14 TiB 0 0 0 up osd.7
8 ssd 13.97069 1.00000 14 TiB 68 MiB 4.8 MiB 12 KiB
63 MiB 14 TiB 0 0 1 up osd.8
9 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB
28 MiB 14 TiB 0 0 0 up osd.9
10 ssd 13.97069 1.00000 14 TiB 32 MiB 4.1 MiB 12 KiB
28 MiB 14 TiB 0 0 0 up osd.10
11 ssd 13.97069 1.00000 14 TiB 68 MiB 4.8 MiB 12 KiB
63 MiB 14 TiB 0 0 1 up osd.11
TOTAL 168 TiB 12 TiB 12 TiB 100 MiB
27 GiB 156 TiB 7.13
So all the date is on osd.1
But I have checked the balancer and seems active:
ceph balancer status
{
"active": true,
"last_optimize_duration": "0:00:00.000368",
"last_optimize_started": "Fri Mar 28 10:55:06 2025",
"mode": "upmap",
"no_optimization_needed": false,
"optimize_result": "Some objects (0.500000) are degraded; try
again later",
"plans": []
}
But the output of the commnad 'ceph config dump|grep balancer' gives
me nothing.
Best,
Mihai
On 2025-03-27 23:06, Anthony D'Atri wrote:
Look at `ceph osd df`. Is the balancer enabled?
On Mar 27, 2025, at 8:50 AM, Mihai Ciubancan
<mihai.ciuban...@eli-np.ro> wrote:
Hello,
My name is Mihai, and I have started using CEPH this mount for a
HPC cluster.
When was lunch in the production the available space shown was
80TB now is 16TB and I didn't do anything, while I'm having 12 OSD
(SSD of 14TB):
sudo ceph osd tree
ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF
-1 167.64825 root default
-3 167.64825 host sto-core-hpc01
0 ssd 13.97069 osd.0 up 1.00000 1.00000
1 ssd 13.97069 osd.1 up 1.00000 1.00000
2 ssd 13.97069 osd.2 up 1.00000 1.00000
3 ssd 13.97069 osd.3 up 1.00000 1.00000
4 ssd 13.97069 osd.4 up 1.00000 1.00000
5 ssd 13.97069 osd.5 up 1.00000 1.00000
6 ssd 13.97069 osd.6 up 1.00000 1.00000
7 ssd 13.97069 osd.7 up 1.00000 1.00000
8 ssd 13.97069 osd.8 up 1.00000 1.00000
9 ssd 13.97069 osd.9 up 1.00000 1.00000
10 ssd 13.97069 osd.10 up 1.00000 1.00000
11 ssd 13.97069 osd.11 up 1.00000 1.00000
sudo ceph df detail
--- RAW STORAGE ---
CLASS SIZE AVAIL USED RAW USED %RAW USED
ssd 168 TiB 156 TiB 12 TiB 12 TiB 7.12
TOTAL 168 TiB 156 TiB 12 TiB 12 TiB 7.12
--- POOLS ---
POOL ID PGS STORED (DATA) (OMAP) OBJECTS
USED (DATA) (OMAP) %USED MAX AVAIL QUOTA OBJECTS QUOTA
BYTES DIRTY USED COMPR UNDER COMPR
.mgr 1 1 705 KiB 705 KiB 0 B 2
1.4 MiB 1.4 MiB 0 B 0 8.1 TiB N/A
N/A N/A 0 B 0 B
cephfs.cephfs.meta 2 16 270 MiB 270 MiB 0 B 85.96k
270 MiB 270 MiB 0 B 0 16 TiB N/A
N/A N/A 0 B 0 B
cephfs.cephfs.data 3 129 12 TiB 12 TiB 0 B 3.73M
12 TiB 12 TiB 0 B 42.49 16 TiB N/A
N/A N/A 0 B 0 B
While on the client side I have this:
$ df -h
10.18.31.1:6789:/ 21T 13T 8.1T 61% /data
I don't know where it's gone all the space that was at the beginning.
Someone has any hint?
Best regards,
Mihai
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-le...@ceph.io
