I like the idea of an expand button. We'll probably use something like that. On the other hand, the primary use case of the Forge quality scores is that of a user trying to evaluate and choose modules, so we're highly focused on that. We probably won't confuse it too much with specific details for the author. HOWEVER.....
By end-of-year, I'm writing a score preview extension for the new PDK as a demo of the plugin system so as a module author, you'll get that feedback during the development phase, even before publishing it. Yay, look at that. A plugin system so easy that a product manager can use it, and a tool that gives you early feedback, right when you need it the most! On Thu, Nov 4, 2021 at 10:14 AM Nick Maludy <nmal...@gmail.com> wrote: > Ben, > > As you describe the summary scores for new users that makes sense. One UI > idea i just had as i read your reply was the following: > > - Give an overall summary score "4.3" or something > - Have a "+" or ">" button that can expand that score and show the > components that went into it > - For each line item show how much each line was worth. This way you can > total up the score. > - Also show the things that were missing and how many points they were > worth. This way module developers know what needs to be fixed/improved in > order to raise their score. > > Just an idea I had and wanted to "brain dump" while it was fresh. > > -Nick > > On Thu, Nov 4, 2021 at 12:56 PM Ben Ford <ben.f...@puppet.com> wrote: > >> On Thu, Nov 4, 2021 at 8:11 AM Gene Liverman <gene.liver...@puppet.com> >> wrote: >> >>> After reading through all the replies, there are bits and pieces from >>> many people that seem to cover most of my opinions. I am going to try and >>> collect those here: >>> >>> What about this idea? Instead of trying to "measure quality" as a >>>> metric, maybe try to expose individual checks or metrics about various >>>> aspects of a module and let the user decide "is this high quality for me, >>>> or not"? >>>> >>> I like this idea. Present a reasonable set of data about things >>> different users may care about, or might should care about, along with a >>> link to some docs explaining why people care about the listed things. >>> >> >> This makes it real easy for Puppet experts to read, but doesn't do much >> for new or casual users. This is why we turned the detail view off >> temporarily on the Forge. Innocuous warnings were unfairly frightening >> users away from decent modules without high scores. Our roadmap for the >> rest of the year includes working on a more user friendly view that will >> reintroduce the details in a more comprehensible way. Some of the score >> explanations are being driven by this very conversation! >> >> >> Regarding unit tests, I find the utilization of rspec-puppet-facts >>> <https://github.com/voxpupuli/rspec-puppet-facts> (and thereby facterdb >>> <https://github.com/voxpupuli/facterdb>) to be a must. I have this >>> opinion for two reasons: >>> 1) as a maintainer, it ensures that my tests work for all the things I >>> have listed in metadata.json (or at least those supported by the gems) >>> which is great in general and especially important when the supported OS >>> lists gets modified. >>> 2) as a user, if I am looking into how a module works it helps me see >>> that the maintainer is testing across all the needed OS's quickly and >>> without having to read every line of every spec test looking for OS combos >>> that I care about. >>> >> >> This is an interesting point. Maybe I'll expand the scope of this just a >> bit to ask a more meta question. If we're programatically assigning a >> quality score, do we think that it's a good idea to give points for >> adhering to a "standard" testing toolchain? eg, puppetlabs_spec_helper, >> facterdb, pdk, etc. >> >> And if we do, then what about modules in which facterdb doesn't actually >> provide any benefits? A module that doesn't use facts doesn't need to test >> with different factsets. How would we distinguish between those cases? >> >> >> As a user, I want to see that there are at least minimal tests covering >>> public bits - aka at least a "it { is_expected.to compile.with_all_deps >>> }" run via rspec-puppet-facts on each supported os. I prefer to see more >>> but also understand that many people who write puppet code are not nearly >>> as comfortable writing tests. >>> >> >> I'm inclined to say that the bar is that a spec file exists for each >> manifest. (Ewoud's use case of defining the tests in a single loop instead >> could be handled by him putting in a file for each path with just a comment >> explaining where the test actually lives, or something similar.) It would >> be better to use rspec-puppet test coverage, but I don't think we're ready >> to actually run the tests on module publication yet. (future improvement?) >> >> >> >>> Regarding integration tests, I love to see them but it takes a lot more >>> knowledge to write them than it does to write a good puppet module. I would >>> love to see straight away that a module has them (and that CI executes >>> them) but wouldn't hold it against an author that they don't have any. >>> >> >> What if the view included a list of platforms the module has acceptance >> tests for. Informational only, rather than affecting the overall quality >> score. This would obviously only know the standard testing toolchain(s), of >> course, but I think this is doable. >> >> >> Personally, I find having a module documented with puppet-strings to be >>> critical for two reasons: >>> 1) it provides lots of useful information within the source code of the >>> module >>> 2) it enables the programmatic generation of a REFERENCE.md file that >>> can then be read on GitHub/GitLab and rendered on the Forge. >>> >>> Examples can also be contained within this and there by be referenced by >>> users in either location too. I think README.md should have a very minimal >>> set of examples in it. Most examples should be kept closer to what they are >>> describing via puppet-strings IMO. >>> >>> Speaking of the README.md, I think looking for select key sections would >>> be worthwhile. I think it should contain the following at a minimum: >>> - an H1 title at the top >>> - badges >>> - that show the version released on the Forge and link to the module >>> on the Forge >>> - build status >>> - license (ideally via the shields.io badge that reads the license >>> file) >>> - an H2 Usage section >>> - an H2 Reference section that contains at least text referencing >>> REFERENCE.md >>> - an H2 Changelog section that at least contains text referring to >>> CHANGELOG.md >>> >> >> Sounds like a puppet-readme-lint tool to me! We can improve the spec >> <https://puppet.com/docs/puppet/latest/modules_documentation.html> and >> test for adherence to it. We could even consider integrating with >> https://errata-ai.github.io/vale-server/docs/style or some such. >> >> >> One other thing I wish there was a good way to flag on, maybe as part of >>> metadata-json-lint, is when author, summary, license, source, project_page, >>> and issues_url are not filled out in an expected format (or absent all >>> together). >>> >> >> We can absolutely improve metadata-lint to include whatever checks we >> think are useful. Probably a good first step would be a formal spec for >> that file 😜 >> >> _._,_._,_ > ------------------------------ > Groups.io Links: > > You receive all messages sent to this group. > > View/Reply Online (#413) <https://groups.io/g/voxpupuli/message/413> | Reply > To Group > <voxpup...@groups.io?subject=Re:%20Re%3A%20%5Bvoxpupuli%5D%20Do%20you%20have%20opinions%20on%20what%20module%20quality%20means%3F> > | Reply To Sender > <nmal...@gmail.com?subject=Private:%20Re:%20Re%3A%20%5Bvoxpupuli%5D%20Do%20you%20have%20opinions%20on%20what%20module%20quality%20means%3F> > | Mute This Topic <https://groups.io/mt/86662086/6177302> | New Topic > <https://groups.io/g/voxpupuli/post> > Your Subscription <https://groups.io/g/voxpupuli/editsub/6177302> | Contact > Group Owner <voxpupuli+ow...@groups.io> | Unsubscribe > <https://groups.io/g/voxpupuli/leave/10372716/6177302/638900109/xyzzy> [ > ben.f...@puppet.com] > _._,_._,_ > > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CACkW_L7DyJGeC-KvR1wRyzevg-5mmZkX06BV5sQBw6UV7kYDQA%40mail.gmail.com.
