Hello, Kirill. Can you, please, write down your proposal? What metrics you want to add in the Ignite?
> 18 февр. 2021 г., в 14:11, ткаленко кирилл <tkalkir...@yandex.ru> написал(а): > > Hi, Nikolay! > > Have we reached a consensus? > > 16.02.2021, 17:09, "ткаленко кирилл" <tkalkir...@yandex.ru>: >> Hi, Zhenya! >> >> Users can also use it, I see nothing wrong with the presence of two metrics. >> >> 16.02.2021, 16:50, "Zhenya Stanilovsky" <arzamas...@mail.ru.invalid>: >>> Kirill, is it good practice to have a metrics for internal use? Don`t >>> think so. >>> +1 witk Nikolay size is more readable than abstract segments count. >>> >>>> Hi, Nikolay! >>>> >>>> For internal use, leave the metric that I propose and also add the >>>> metric: Count of bytes logged in WAL. Why not "written" because for the >>>> mmap we cannot track when the physical writting will occur. >>>> >>>> 16.02.2021, 15:42, "Nikolay Izhikov" < nizhi...@apache.org >: >>>>> Kirill. >>>>> >>>>> «Count of segments» is a very internal thing for a regular user. >>>>> Regular user don’t want to know about such things. >>>>> >>>>> You suggest to calculate the number (space required to store WAL) with >>>>> some kind of rough calculation, and with the «Count of bytes written in >>>>> WAL» we can have exact number without any suggestions or calculations. >>>>> >>>>> Moreover, «Count of bytes written in WAL» is independent on internal >>>>> WAL implementation. >>>>> >>>>> So, I think exact number is always better to have then some >>>>> approximation. >>>>> >>>>> What do you think? >>>>> >>>>>> 15 февр. 2021 г., в 20:45, ткаленко кирилл < tkalkir...@yandex.ru > >>>>>> написал(а): >>>>>> >>>>>> Hi, Nikolay! >>>>>> >>>>>> We set the number of segments in the working directory, we also >>>>>> delete by segment, it seems that this is a matter of usability. I prefer >>>>>> to dwell on my own version, this is a simple metric that does not hurt >>>>>> and you can add more as needed. >>>>>> >>>>>> 15.02.2021, 17:10, "Nikolay Izhikov" < nizhi...@apache.org >: >>>>>>> My suggestion that «count of files» is meaningless number. >>>>>>> And «count of bytes written to the files» is useful number to know >>>>>>> and use for capacity planning.. >>>>>>> >>>>>>>> 15 февр. 2021 г., в 15:59, ткаленко кирилл < tkalkir...@yandex.ru >>>>>>>> > написал(а): >>>>>>>> >>>>>>>> Hi, Nikolay! >>>>>>>> >>>>>>>> There may be a number (count of segments * segment size) or there >>>>>>>> may be a count of segments, whichever is more convenient for the user. >>>>>>>> >>>>>>>> 15.02.2021, 13:14, "Nikolay Izhikov" < nizhi...@apache.org >: >>>>>>>>> Hello, Kirill. >>>>>>>>> >>>>>>>>> Thanks for an answers. >>>>>>>>> Now, I understand your intentions. >>>>>>>>> >>>>>>>>>> t also seems that it will be more natural to operate not just >>>>>>>>>> bytes but multiples of a segment. >>>>>>>>> >>>>>>>>> Can’t agree here. >>>>>>>>> From my point of view - it’s better to know exact number, not >>>>>>>>> just «count of segments». >>>>>>>>> >>>>>>>>>> 15 февр. 2021 г., в 13:00, ткаленко кирилл < >>>>>>>>>> tkalkir...@yandex.ru > написал(а): >>>>>>>>>> >>>>>>>>>> Hello, Nikolay! >>>>>>>>>> >>>>>>>>>> The period of one day (24h) seems more natural, you can take >>>>>>>>>> more or less, I think that one day may not be enough, and it is >>>>>>>>>> worth getting the metric for several days (collect statistics) for >>>>>>>>>> example a week. Yes, the total size of the segments may not be >>>>>>>>>> DataStorageConfiguration#getMaxWalArchiveSize, but for capacity >>>>>>>>>> planning, accuracy is not so important to us, since the load can >>>>>>>>>> always change, it will hurt users more if we overflow the archive >>>>>>>>>> and it will not be able to start the node. So to say that more is >>>>>>>>>> better than less, it also seems that it will be more natural to >>>>>>>>>> operate not just bytes but multiples of a segment. >>>>>>>>>> >>>>>>>>>> In separate threads, you can discuss the metric that you >>>>>>>>>> propose about page memory and indexes estimates. >>>>>>>>>> >>>>>>>>>> 14.02.2021, 11:54, "Nikolay Izhikov" < nizhi...@apache.org >: >>>>>>>>>>> Hello, Kirill >>>>>>>>>>> >>>>>>>>>>> Your conclusions still not clear for me. >>>>>>>>>>> >>>>>>>>>>>> It is not possible for us to estimate how much space a user >>>>>>>>>>>> will need in the archive so as not to overflow it under its load >>>>>>>>>>>> We take the maximum 44 and multiply it by a >>>>>>>>>>>> DataStorageConfiguration#getWalSegmentSize >>>>>>>>>>> >>>>>>>>>>> Why you take a single day (24h) for a standard period? Is >>>>>>>>>>> there any rationale behind this? >>>>>>>>>>> >>>>>>>>>>> 1. We have `walAutoArchiveAfterInactivity` property. So WAL >>>>>>>>>>> segment can have a size less than the maximum. >>>>>>>>>>> 2. For CDC feature I want to introduce «WAL force rollover >>>>>>>>>>> timeout» to make data available for a consumer in a guaranteed >>>>>>>>>>> period [1]. >>>>>>>>>>> >>>>>>>>>>> Why does the user want to estimate those numbers in the first >>>>>>>>>>> place? >>>>>>>>>>> Are we talking about some kind of capacity planning? >>>>>>>>>>> >>>>>>>>>>> If yes, then maybe it will be better to have a metric for a >>>>>>>>>>> count of bytes written in the WAL? >>>>>>>>>>> With it, we will have an exact number of space we need for WAL. >>>>>>>>>>> >>>>>>>>>>> How user should estimate capacity for a page memory and >>>>>>>>>>> indexes? >>>>>>>>>>> >>>>>>>>>>> [1] https://issues.apache.org/jira/browse/IGNITE-13582 >>>>>>>>>>> >>>>>>>>>>>> 14 февр. 2021 г., в 09:48, ткаленко кирилл < >>>>>>>>>>>> tkalkir...@yandex.ru > написал(а): >>>>>>>>>>>> >>>>>>>>>>>> Hi, Nikolay! >>>>>>>>>>>> >>>>>>>>>>>> The user will be able to take the >>>>>>>>>>>> getLastArchivedSegmentIndex every day and remember it and do it, >>>>>>>>>>>> say, for several days. >>>>>>>>>>>> >>>>>>>>>>>> For example, when starting the application, the >>>>>>>>>>>> getLastArchivedSegmentIndex is 0, then at the end of the first day >>>>>>>>>>>> the value will be 30 at the end of the second 55 and at the end of >>>>>>>>>>>> the third 99. >>>>>>>>>>>> It turns out that 30 segments were used for the first day, >>>>>>>>>>>> 25 for the second and 44 for the third. We take the maximum 44 and >>>>>>>>>>>> multiply it by a DataStorageConfiguration#getWalSegmentSize, and >>>>>>>>>>>> we get the possible maximum that the archive overflow was the >>>>>>>>>>>> least likely. If the user uses compression, then it can be >>>>>>>>>>>> subtracted from the result (result * >>>>>>>>>>>> getMaxSizeCompressedArchivedSegment). >>>>>>>>>>>> >>>>>>>>>>>> 13.02.2021, 10:47, "Nikolay Izhikov" < nizhi...@apache.org >: >>>>>>>>>>>>> Hello, Kirill. >>>>>>>>>>>>> >>>>>>>>>>>>>> It is not possible for us to estimate how much space a >>>>>>>>>>>>>> user will need in the archive so as not to overflow it under its >>>>>>>>>>>>>> load >>>>>>>>>>>>> >>>>>>>>>>>>> It still not clear for me why do we need those metrics. >>>>>>>>>>>>> Can you please, write down specific scenario - how user >>>>>>>>>>>>> will use these metrics to estimate required WAL volume? >>>>>>>>>>>>> >>>>>>>>>>>>>> 12 февр. 2021 г., в 19:35, ткаленко кирилл < >>>>>>>>>>>>>> tkalkir...@yandex.ru > написал(а): >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, Nikolay! >>>>>>>>>>>>>> >>>>>>>>>>>>>> It is not possible for us to estimate how much space a >>>>>>>>>>>>>> user will need in the archive so as not to overflow it under its >>>>>>>>>>>>>> load. And the proposed metrics will allow you to make a rough >>>>>>>>>>>>>> estimate. >>>>>>>>>>>>>> >>>>>>>>>>>>>> 12.02.2021, 17:23, "Nikolay Izhikov" < >>>>>>>>>>>>>> nizhi...@apache.org >: >>>>>>>>>>>>>>> Hello, Kirill. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Can you, please, clarify - What question about WAL user >>>>>>>>>>>>>>> have in mind? >>>>>>>>>>>>>>> And what answers he(or she) gets with these new metrics? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 12 февр. 2021 г., в 14:26, ткаленко кирилл < >>>>>>>>>>>>>>>> tkalkir...@yandex.ru > написал(а): >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi everyone! >>>>>>>>>>>>>>>> At the moment, I have not found an opportunity to >>>>>>>>>>>>>>>> estimate how many WAL segments fall into the archive, say per >>>>>>>>>>>>>>>> day. >>>>>>>>>>>>>>>> So I created a ticket >>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/IGNITE-14170 to add a >>>>>>>>>>>>>>>> couple of new metrics.
