Hi again I have started with a document that clarify the severity levels. I also introduced the level "critial" but I'm not sure it adds any value.
https://inguza.com/document/debian-security-severity-levels This is just a first draft. It is not final. But comments are welcome. // Ola On Wed, 10 Apr 2024 at 22:03, Ola Lundqvist <o...@inguza.com> wrote: > > Hi Raphael > > You can see corrected statistics in a separate email. > > Now to comment a few things below. > > On Wed, 10 Apr 2024 at 10:49, Raphael Hertzog <hert...@debian.org> wrote: > > > > Hello, > > > > On Tue, 09 Apr 2024, Ola Lundqvist wrote: > > > Let me use some data from CVEs for last year 2023. > > > I used the following method to extract the data > > > grep -B 5 '\[buster\]' list | grep -A 5 "^CVE-2023-" | grep '\[buster\]' > > > and then grepped for the end-of-life, not-affected (and so on to count > > > further). > > > It is not a perfect method, but I think it is good enough to crunch the > > > numbers. > > > > > > 1260 CVEs with tag [buster] > > > 382 end-of-life > > > 410 not-affected > > > 281 no-dsa > > > 71 postponed > > > 58 ignored > > > 58 fixed (most in the linux kernel) > > > > Those numbers are quite surprising. I hope there's some error somewhere > > otherwise I wonder what has been done in the 2400+ hours paid each year to > > work on LTS... I'm pretty sure we have fixed more than 58 CVE. The average > > month has 20 to 30 updates (see > > https://lists.debian.org/debian-lts-announce/2024/03/threads.html for > > example). > > Yes I was surprised as well. When I checked yesterday I could not find > the fault but today with fresh eyes it was rather obvious. :-) > Sorry for the confusion. See the other email about that. > > > > A large portion of the CVEs are marked as no-dsa for a long time. We > > > still have 200 of them from year 2019 for example. > > > > > > And I think this is a good practice. Security is always a balance > > > between a lot of things. Fixing everything will likely result in a lot > > > of tools breaking. > > > Also following the regular Debian Security team is also a good > > > practice. They do a good job of analyzing things and we should > > > normally not fix issues in buster when it will not be fixed in later > > > releases. So this makes a lot of sense. > > > > This comment still conflates "no-dsa" with "doesn't need to be fixed". We > > clearly told you many times that it doesn't mean that at all. > > Did I write that? I cannot see it it what you comment. But after > reading the whole email I think I understand why you think so. > If you read my proposal I suggest we postpone all no-dsa and then wait > for point release updates as one way to handle it. > That is basically the same thing as what you write below, that no-dsa > means no need for an urgent fix. > > But then you commented that we should do our own judgement. Yes it > would be good if we have a guilde. More about that below. > > > "no-dsa" means "doesn't need to be urgently fixed, and will not be fixed > > by the security team". It basically defers the question to the package > > maintainer. It's up to the package maintainer to decides between > > "to fix via spu", "postponed" or "ignored". > > I agree to this. > > > Some package maintainers will typically decide to fix it via a point > > release. But they rarely update the triaging to document "postponed" or > > "ignored". So that's why it's up to the LTS team to make that call > > when we are (alone) in charge of a given release. > > This is a good point. I had missed that package maintainers do not > update the security tracker. Can we have tools to help us with this? > > > And regarding "we should normally not fix issues in buster when it will > > not be fixed in later releases", we have already explained that it goes > > the other way around: if we decide that something is worth to be fixed > > in buster, then we should help to get it fixed in later releases. > > I do agree with that approach. The question then is when we should do that. > I think we need some type of guidance. It is very easy to fall into extremes. > One extreme is to only follow secteam. > The other extreme is to try to fix all no-dsa no matter how unimportant it is. > > > Obviously if the sec-team triages it as unimportant/ignored, then we are > > unlikely to fix it. But when the sec team opts for no-dsa, then we are > > entitled to decide to fix it and we can also prepare the SPU to fix newer > > releases. > > Yes, then the question is what rule/guide to follow. We can't fix all. > That is clear from the statistics. > > > > 2) Another way is to use the definition we have of > > > unimportant/low/medium/high classification and use that as judgement. > > > > I wanted to comment about this classification. While it's not in active > > use by the security team, I think it would be nice if we could provide > > such a classification: > > Yes, I think so. Or some other method we choose to follow. > But I think we need to clarify it, because it is a little fuźzy at the moment. > > > 1/ it can help to drive attention to the the most pressing CVE > > 2/ it lets us monitor how well we are performing for different severities > > (in terms of delay to provide a fix) > > > > Clearly the LTS sponsors who are funding security work would like to see > > some things like "90% of the high-priority CVE are fixed within one week". > > But we are not able to provide any data. > > Precisely. > > > Also we are regularly telling customers/sponsors that we are doing on own > > assessment of the importance of CVE and that it may diverge from the > > standard CVSS score that they use as reference. In the ideal world, we > > would provide our own CVSS score with explanations but we are quite far > > from this. In the mean time, it would already be great if we had more > > reliable low/medium/high scoring... > > Yes it would be really good. If we add this as part of the triaging > work it would be good. > In a way we need to do that already because we say that we should not > simply tag no-dsa, but rather postpone or not. > > There are a few things we need to decide on: > - What severity levels warrants the package to be in dla-needed? Is > low one of them? > - Should all medium be in dla-needed? > > If you ask me I think the line is somewhere in medium, but I'm not > sure of what definition to use yet. > > Cheers > > // Ola > > > > > > Cheers, > > -- > > ⢀⣴⠾⠻⢶⣦⠀ Raphaël Hertzog <hert...@debian.org> > > ⣾⠁⢠⠒⠀⣿⡁ > > ⢿⡄⠘⠷⠚⠋ The Debian Handbook: https://debian-handbook.info/get/ > > ⠈⠳⣄⠀⠀⠀⠀ Debian Long Term Support: https://deb.li/LTS > > > > -- > --- Inguza Technology AB --- MSc in Information Technology ---- > | o...@inguza.com o...@debian.org | > | http://inguza.com/ Mobile: +46 (0)70-332 1551 | > --------------------------------------------------------------- -- --- Inguza Technology AB --- MSc in Information Technology ---- | o...@inguza.com o...@debian.org | | http://inguza.com/ Mobile: +46 (0)70-332 1551 | ---------------------------------------------------------------
