IIRC Prometheus WAL no longer fsyncs on a tight interval. It simply writes to page cache, so any flushing is actually controlled by the kernel.
sysctl vm.dirty_writeback_centisecs The only forced syncs you'll see happen at compaction time, every 2 hours. See prometheus_tsdb_wal_fsync_duration_seconds_count. So there's no need to do any of the shenanigans that VM does. On Mon, Mar 13, 2023 at 9:49 AM Brian Candler <[email protected]> wrote: > Depending on other constraints you have, you could run prometheus in agent > mode, with its WAL on ramdisk, doing remote_write to some other system. > > I also remember reading recently about a change in VictoriaMetrics to > defer the flushing of memory to reduce SSD wear: > https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.85.0 > > https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337#issuecomment-1338482663 > > On Sunday, 12 March 2023 at 23:18:48 UTC Julien Pivotto wrote: > >> Hello, >> >> It's not possible directly. >> >> One possible workaround that comes to mind is using snapshots. >> >> You could take periodic snapshots of the Prometheus database and flush >> them to the filesystem at a lower frequency, say every 5 minutes, as you >> mentioned. >> >> Then you can sync the snapshot to disk. >> >> Note: You can decide to snapshot the head or just the blocks. >> >> On 12 Mar 15:18, Marian Velez wrote: >> > Hi! >> > I need a hand trying to delay I/O flush operations into the >> hw/filesystem, >> > due to poor hardware SSD lifetime. >> > I'm at this time unable to replace the SSD since I'm at a very remote >> > location, so in term I was thinking on extending it as match as >> possible by >> > flushing prometheus synchronous I/O operations into the filesystem like >> > every 5 minutes or so. >> > >> > Initially I was thinking on doing some kind of hybrid thing in which I >> run >> > tmpfs and sync it into the proper FS, but the DB keep breaking due to >> open >> > files, which is kind of expected. >> > >> > On the other hand, I couldn't find any custom Linux filesystem, fuse or >> > not, that would allow me to control the underlying FS flushing >> frequency, >> > so I was trying to see if I had the ability to do that on the >> prometheus DB >> > side. >> > >> > Do you have any clue if this is achievable? >> > >> > Thanks in advance! >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups "Prometheus Users" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/4ca96afd-e76d-407c-9006-4e7320bcae74n%40googlegroups.com. >> >> >> >> -- >> Julien Pivotto >> @roidelapluie >> > -- > You received this message because you are subscribed to the Google Groups > "Prometheus Users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/prometheus-users/050e54c1-0aee-44ea-b805-57dc60b48a55n%40googlegroups.com > <https://groups.google.com/d/msgid/prometheus-users/050e54c1-0aee-44ea-b805-57dc60b48a55n%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/CABbyFmpSK9czJhOe-f2BZZvPjV7Kh_Y_X9hqqszi%2B8WWCsFQ8A%40mail.gmail.com.
