When enforcing either max-disk-usage or min-disk-free the broker looks at the *whole* disk so that any data written to disk is considered in the calculation. This, of course, applies to files in the broker's journal. It also applies to data written by other processes. For example, if there was another process on the machine filling up the disk then the broker would see that and potentially shut itself down if the max-disk-usage or min-disk-free settings were breached.
To be clear, journal files are pre-allocated. In other words, even an "empty" journal will take up space on disk. Therefore, when messages are written to the journal they won't occupy additional space. The new messages simply replace whatever bytes existed in the file previously. The only exception here is when the overall size of the journal exceeds the existing pool of files at which point the broker must allocate another journal file. This new file will, of course, occupy additional disk space. Hope that helps! Justin On Mon, Mar 4, 2024 at 11:41 AM andrea bisogno <bisoma...@hotmail.it> wrote: > Hi Justin, > thanks for your reply. > > There is just a latest point I would need to clarify. > Do the "max-disk-usage"/"min-disk-free" settings apply also to the > persisted messages, and not only to the paging behavior, right? > I will try to explain my test scenario a little more: > > * > we are working with the broker in the k8s scenario (broker + operator), > where we enabled the persistence > * > this resulted into a "max-disk-usage" set by default to 90 on the > broker.xml file and a persistence volume of 2Gi reserved by default for the > persistence > * > we are simulating some MQTT messages sent with qos 2, while some > subscribers (that subscribed to the corresponding addresses with qos2) are > disconnected; > * > since the clients are disconnected, we observed that the messages sent > were immediately stored in the "/opt/amq-broker/data/journal" folder, as > expected (waiting for the subscribers to reconnect again) > * > so, we would like to understand if this persistence behavior, even without > passing for paging, can be controlled with the "max-disk-usage" that you > mentioned before, to make sure we don't saturate the 2Gi of the persistence > volume > > Many thanks for your help > ________________________________ > Da: Justin Bertram <jbert...@apache.org> > Inviato: venerdì 1 marzo 2024 15:36 > A: users@activemq.apache.org <users@activemq.apache.org> > Oggetto: Re: Persistence: is there way to limit the maximum storage size? > > The journal exists in order to restore the persistent messages in the > broker's memory when it restarts. Therefore, the amount of data in the > journal is directly proportional to the amount of data in memory. > Therefore, if you want to control the amount of data in the journal you can > do so by controlling the amount of data in memory and to do that you > configure the size of your addresses (e.g. per-address via max-size-bytes > [1] or max-size-messages [1] or globally via global-max-size [2] or > global-max-messages [3]). When the address gets full the > address-full-policy is enforced which might also result in writing data to > disk (i.e. if using PAGE). In order to prevent the disk from filling up you > can use max-disk-usage [4] or min-disk-free [5]. > > > Justin > > [1] > > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fpaging.html%23configuration-2&data=05%7C02%7C%7C2de9c226f3f447fd6c8408dc3a05b599%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638449043388777235%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=MMc1VrmKvNJTRptznK3RbB%2FOiHZ1dKvj1i2tY30PaaE%3D&reserved=0 > < > https://activemq.apache.org/components/artemis/documentation/latest/paging.html#configuration-2 > > > [2] > > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fpaging.html%23global-max-size&data=05%7C02%7C%7C2de9c226f3f447fd6c8408dc3a05b599%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638449043388785214%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=fYEseAt4%2B7zNBRFHw%2BcSxRhN%2B2a4OQzbqNKL3%2FEAZio%3D&reserved=0 > < > https://activemq.apache.org/components/artemis/documentation/latest/paging.html#global-max-size > > > [3] > > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fpaging.html%23global-max-messages&data=05%7C02%7C%7C2de9c226f3f447fd6c8408dc3a05b599%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638449043388790822%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=%2B76rrOHgiHLFDTVnEJgxKHTAcxNJcK0YuKPNZ23DZtg%3D&reserved=0 > < > https://activemq.apache.org/components/artemis/documentation/latest/paging.html#global-max-messages > > > [4] > > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fpaging.html%23max-disk-usage&data=05%7C02%7C%7C2de9c226f3f447fd6c8408dc3a05b599%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638449043388795441%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=nrO53iiWjCUH9i2Lm83kKLh8Q4zpUdXmydSX9oCR5P0%3D&reserved=0 > < > https://activemq.apache.org/components/artemis/documentation/latest/paging.html#max-disk-usage > > > [5] > > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fpaging.html%23minimum-disk-free&data=05%7C02%7C%7C2de9c226f3f447fd6c8408dc3a05b599%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638449043388799925%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=rPySymSBCgyAW4%2Fb%2FTwriUGEqxbxCdBCGCcitiayzRo%3D&reserved=0 > < > https://activemq.apache.org/components/artemis/documentation/latest/paging.html#minimum-disk-free > > > > On Fri, Mar 1, 2024 at 5:05 AM andrea bisogno <bisoma...@hotmail.it> > wrote: > > > Hi support, > > I would need some insights about the Artemis Broker Persistence feature. > > Looking into the documentation at > > > https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Factivemq.apache.org%2Fcomponents%2Fartemis%2Fdocumentation%2Flatest%2Fpersistence.html&data=05%7C02%7C%7C2de9c226f3f447fd6c8408dc3a05b599%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C638449043388804379%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=GxzAhBRvY2pm4cnSHZAwX81lLGIiZyc5WPvAC3r1Y0M%3D&reserved=0 > < > https://activemq.apache.org/components/artemis/documentation/latest/persistence.html > >, > > I was not able to find anything about possible limits to disk size usage > > once te File Journal persistence option is enabled. > > We would like to understand if (and how) it is possible to limit the > > number of journal files produced, and so be able to limit the disk size > > used by the persistence. > > Is it possible to configure, for example, a maximum numer of Journal > > files, and make the broker reuse the first one (i.e. the oldest file) > once > > the max number of files is reached, overwriting the oldest data? > > Basically, it would be important for us being able to avoid disk > > saturation, to avoid dangerous situation where the persistence can cause > > unexpected issues due to too much storage space used. > > > > Many thanks in advance for your support, > > > > Andrea > > >
