Thanks for the response Brain, it was 175GB and with the last restart, I increased it to 500 GB, sorry I did a typo there.
It's not generating that much data in 2 ~ 4 hrs, but it is doing over a week. The data is stored on the local filesystem ext4. not on nas. # ls 01FTRRKBEQV1QX0HAM2MXG1W3Q 01FTTWSEGZ4MQN3NXAR12R8GSD 01FTTX6H5GM758FVPHK1DTSH9S chunks_head lock queries.active wal # ls chunks_head/ 000175 000176 I don't see any errors in the logs, but still checking on the issue. Thanks On Tuesday, February 1, 2022 at 6:37:29 AM UTC-5 Brian Candler wrote: > I am having issues with chunks_head directory. I have the tsdb retention >> size to 175 GB >> > ... > >> --storage.tsdb.retention.size=500GB >> > > That doesn't appear to be consistent with what you said above. > > However, in any case, the "chunks_head" directory should only contain data > created in the last 2-4 hours, before it gets moved to its final location. > > Is it possible that you're generating 689GB of data in 2-4 hours? That's > a *lot* of metrics, and it seems unlikely - but if so, this would be > expected behaviour. > > If not, then there's some problem moving the head chunks into their final > location. Your storage directory should look something like this (from > v2.30.3): > > root@prometheus:~# ls /var/lib/prometheus/data/ > 01FRDMW4G1T19JM989NE1VKZ32 01FSGD7TFJEWA6FJ08PJ2TZZPG > 01FTK5KH12W0RDWZA85NSTP4TX chunks_head > 01FRKE8RPDW2ZYSJFXEDD7QK3S 01FSP6MG73NQQ996C7EWH49MQF > 01FTRZ05RH3C8RKJ4PQ162QJT8 lock > 01FRS7NBJST1MW7R3XDBE9592M 01FSW012JNK10VV5WWZ4C2H3V9 > 01FTSKK60DX4D8C908BYHZTR8F queries.active > 01FRZ120RQQZJW24T3BVP9NFTT 01FT1SDN5R9JYEQFQ90JCS2Y2Z > 01FTT869E3DW1V0QYSPZ921CKQ wal > 01FS4TEKSMZ3NJ2M6NNEAH9K63 01FT7JTABCH3D1JDQYZAPJ9QXK > 01FTT86AK2N6VA00RSCYB9AC6J > 01FSAKV7VSS6VMPDCJAWNZGT6V 01FTDC6WGMY25WCQ3JTKZECGPX > 01FTTF20P47NVE06W79PKKFXFJ > > The majority of storage will be in those other directories, which are > created automatically, and are what the retention setting will control. If > this isn't happening, then you have a problem. > > I suggest you check the log output from prometheus for errors (e.g. > "journalctl -eu prometheus" if you're running it under systemd). It could > be that it's failing to create these directories, and data is sticking > around in the head. Are permissions on /apps/prometheus/prometheus_data > correct? Are you storing data on a block filesystem (e.g. local disk or a > filesystem in Amazon EBS)? Prometheus does not support writing data to any > sort of NAS filesystem, e.g. NFS / Samba / EFS. > > There were a couple of minor fixes in v2.31.1 and v2.31.2, but I don't > think they are relevant here. > > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/b8c0105d-6ba3-4e26-916d-bf263822c0b7n%40googlegroups.com.
