> > I am having issues with chunks_head directory. I have the tsdb retention > size to 175 GB > ...
> --storage.tsdb.retention.size=500GB > That doesn't appear to be consistent with what you said above. However, in any case, the "chunks_head" directory should only contain data created in the last 2-4 hours, before it gets moved to its final location. Is it possible that you're generating 689GB of data in 2-4 hours? That's a *lot* of metrics, and it seems unlikely - but if so, this would be expected behaviour. If not, then there's some problem moving the head chunks into their final location. Your storage directory should look something like this (from v2.30.3): root@prometheus:~# ls /var/lib/prometheus/data/ 01FRDMW4G1T19JM989NE1VKZ32 01FSGD7TFJEWA6FJ08PJ2TZZPG 01FTK5KH12W0RDWZA85NSTP4TX chunks_head 01FRKE8RPDW2ZYSJFXEDD7QK3S 01FSP6MG73NQQ996C7EWH49MQF 01FTRZ05RH3C8RKJ4PQ162QJT8 lock 01FRS7NBJST1MW7R3XDBE9592M 01FSW012JNK10VV5WWZ4C2H3V9 01FTSKK60DX4D8C908BYHZTR8F queries.active 01FRZ120RQQZJW24T3BVP9NFTT 01FT1SDN5R9JYEQFQ90JCS2Y2Z 01FTT869E3DW1V0QYSPZ921CKQ wal 01FS4TEKSMZ3NJ2M6NNEAH9K63 01FT7JTABCH3D1JDQYZAPJ9QXK 01FTT86AK2N6VA00RSCYB9AC6J 01FSAKV7VSS6VMPDCJAWNZGT6V 01FTDC6WGMY25WCQ3JTKZECGPX 01FTTF20P47NVE06W79PKKFXFJ The majority of storage will be in those other directories, which are created automatically, and are what the retention setting will control. If this isn't happening, then you have a problem. I suggest you check the log output from prometheus for errors (e.g. "journalctl -eu prometheus" if you're running it under systemd). It could be that it's failing to create these directories, and data is sticking around in the head. Are permissions on /apps/prometheus/prometheus_data correct? Are you storing data on a block filesystem (e.g. local disk or a filesystem in Amazon EBS)? Prometheus does not support writing data to any sort of NAS filesystem, e.g. NFS / Samba / EFS. There were a couple of minor fixes in v2.31.1 and v2.31.2, but I don't think they are relevant here. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/07a22d78-d8f3-4421-99c6-fcf61cf38dc7n%40googlegroups.com.
