Interesting. Indeed, if I graph prometheus_tsdb_wal_fsync_duration_seconds_count I see it only increment once every 2 hours (on a small test system with v2.37.6)
On Monday, 13 March 2023 at 09:42:52 UTC Ben Kochie wrote: > IIRC Prometheus WAL no longer fsyncs on a tight interval. It simply writes > to page cache, so any flushing is actually controlled by the kernel. > > sysctl vm.dirty_writeback_centisecs > > The only forced syncs you'll see happen at compaction time, every 2 hours. > See prometheus_tsdb_wal_fsync_duration_seconds_count. > > So there's no need to do any of the shenanigans that VM does. > > On Mon, Mar 13, 2023 at 9:49 AM Brian Candler <[email protected]> wrote: > >> Depending on other constraints you have, you could run prometheus in >> agent mode, with its WAL on ramdisk, doing remote_write to some other >> system. >> >> I also remember reading recently about a change in VictoriaMetrics to >> defer the flushing of memory to reduce SSD wear: >> https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.85.0 >> >> https://github.com/VictoriaMetrics/VictoriaMetrics/issues/3337#issuecomment-1338482663 >> >> On Sunday, 12 March 2023 at 23:18:48 UTC Julien Pivotto wrote: >> >>> Hello, >>> >>> It's not possible directly. >>> >>> One possible workaround that comes to mind is using snapshots. >>> >>> You could take periodic snapshots of the Prometheus database and flush >>> them to the filesystem at a lower frequency, say every 5 minutes, as you >>> mentioned. >>> >>> Then you can sync the snapshot to disk. >>> >>> Note: You can decide to snapshot the head or just the blocks. >>> >>> On 12 Mar 15:18, Marian Velez wrote: >>> > Hi! >>> > I need a hand trying to delay I/O flush operations into the >>> hw/filesystem, >>> > due to poor hardware SSD lifetime. >>> > I'm at this time unable to replace the SSD since I'm at a very remote >>> > location, so in term I was thinking on extending it as match as >>> possible by >>> > flushing prometheus synchronous I/O operations into the filesystem >>> like >>> > every 5 minutes or so. >>> > >>> > Initially I was thinking on doing some kind of hybrid thing in which I >>> run >>> > tmpfs and sync it into the proper FS, but the DB keep breaking due to >>> open >>> > files, which is kind of expected. >>> > >>> > On the other hand, I couldn't find any custom Linux filesystem, fuse >>> or >>> > not, that would allow me to control the underlying FS flushing >>> frequency, >>> > so I was trying to see if I had the ability to do that on the >>> prometheus DB >>> > side. >>> > >>> > Do you have any clue if this is achievable? >>> > >>> > Thanks in advance! >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> Groups "Prometheus Users" group. >>> > To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> > To view this discussion on the web visit >>> https://groups.google.com/d/msgid/prometheus-users/4ca96afd-e76d-407c-9006-4e7320bcae74n%40googlegroups.com. >>> >>> >>> >>> >>> -- >>> Julien Pivotto >>> @roidelapluie >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Prometheus Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> > To view this discussion on the web visit >> https://groups.google.com/d/msgid/prometheus-users/050e54c1-0aee-44ea-b805-57dc60b48a55n%40googlegroups.com >> >> <https://groups.google.com/d/msgid/prometheus-users/050e54c1-0aee-44ea-b805-57dc60b48a55n%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/66eb8c59-b869-42a5-b39f-02b88cae7159n%40googlegroups.com.
