On Thu, 2011-01-20 at 05:54 -0500, Samuel Greenfeld wrote: > Unfortunately while I am familiar with a few test case management > systems, I have not been involved with the Fedora project long enough > to know its workflows. And a quick search online is not turning up > much about Nitrate, with its changelog[1] implying it was first made > open source software in July 2010. > > I might be able to provide some general comments though if I knew more > about the product. So for those of us unfamiliar with the history of > Nitrate, could you please answer the following:
Really good list of questions Samuel, thanks for jumping in. I know Hurry can provide feedback on the questions. However, I'll add my thoughts as well. > 1. What is the history of Nitrate and the Fedora Project? What > does the Fedora project expect to gain from using it? Hurry can touch on the goals for using nitrate. As for history, I can add my experiences ... This goes back to an eval we did using testopia in Fedora many releases agove. Unfortunately, the effort was canceled due to license incompatibility between Fedora and testopia. At that point, we invested in leveraging the wiki to best of our ability. https://fedoraproject.org/wiki/QA/Testopia_Evaluation https://bugzilla.redhat.com/show_bug.cgi?id=450013 After this, several folks got together and decided they would implement a new front-end on top of the testopia db schema. This would resolve the original license incompatibility and address usability issues that were raised with the testopia UI. The project started development internally, and was open-sourced in 2010. > 1. Is a sample play/sandbox test instance with more-or-less full > access available online? Not yet, I believe that's in plan for sometime during Fedora 15. Hurry and I were discussing the requirements for such an instance earlier this week. If you're interested in helping here, just let us know :) > 1. How does Nitrate compare to other open & closed sourced TCMS > solutions? Why was it written as opposed to using an existing > solution, and what are its straights & weaknesses? See history comments above. Also, maybe the nitrate developers [1] can offer more insight on how it compares to other open-source solutions? I *think* that comparison work has been done in the past, I'm just not sure where to find the results. [1] https://fedorahosted.org/mailman/listinfo/nitrate-devel > 1. Is test case & test plan import and export (to XML, etc.) > support available? If so is this compatible with any other > TCMS's import/export system? I believe import/export is supported using the testopia.dtd format. http://git.fedorahosted.org/git/?p=nitrate.git;a=blob;f=trunk/nitrate/docs/ChangeLog - Fixed #564258 - [REF] Ability to export/print specified cases - Fixed #612803 - add an export feature for test case runs, can export … > 1. Are nested test plan and/or test case hierarchies supported? I don't recall. Perhaps Hurry or the nitrate developers know? I know this feature has been discussed a *lot* with nitrate, and other solutions we've used in the past. I don't think support for nested test plans is something we've had a tremendous need for now, so I don't anticipate this being a MUSTHAVE feature in the near term. > 1. Can multiple projects share test cases, and even reference > older versions of test cases if they are lagging behind the > current rawhide/Fedora release? Will Spins be able to make > their own (simultaneously running) test plans? This is the hope. It's not really useful if we can only use it for release validation. I don't think we've fully explored how best to model other spins/projects, but I don't foresee big problems there. That will be fun to explore on the sandbox/staging instance. With regards to referencing older versions of a test case, I believe that support is there, although I'm not certain it's right for our needs. Keeping test documentation (plans and cases) updated is a pretty sizable maintenance challenge. I've seen many instances where support for versioned test cases allows test plans to suffer over time as they were linked against old and inaccurate test cases. Much like how the wiki is used now, we have support for linking against older versions of tests (wiki history), but we rarely ever use that feature. I expect that trend would continue in the short-term. > 1. How long will historical test case results be made available? I suspect the limiting factor here is database size. I'm not aware of any rules or process that would require removal/archival of old results. However, at some point that could certainly be an issue we'd need to plan for. > 1. Is there any plan to tie this into Bodhi and other tools to > detect updated packages that may imply test cases need > re-running and/or updating? That is certainly possible, but there are currently no detailed plans to integrate bodhi/f-e-k with nitrate specifically. Nitrate offers a robust XMLRPC interface to allow other tools, such as bodhi, to query for applicable tests/results/runs etc... So I wouldn't think it would be a difficult task. That said, Adam Williamson is exploring this linkage using our current wiki-based test management system. While the two implementations will be completely different, the end result is the same ... improved integration between our update tools (bodhi/f-e-k) and our test infrastructure. > 1. Is this going to be available as a Fedorahosted service like > Trac is? If so will all the instances be able to share test > cases? Hmm, I don't envision multiple hosted instances of nitrate (or any other test case management) in Fedora. I expect we would have a single test management system available for all to use, much like we have a build system, updates and bug reporting. > 1. Is there any concern that changing test tracking systems may > encourage/discourage existing testers to participate? Yes! That's part of why Hurry is investing a lot of time researching our current workflows and comparing features between our current wiki-based implementation. > Some of this information may be useful to post on the Trac main page > and/or in the Fedora wiki. > > Thanks in advance for your time. Thank you for your input, good comments/questions! Thanks, James
signature.asc
Description: This is a digitally signed message part
-- test mailing list test@lists.fedoraproject.org To unsubscribe: https://admin.fedoraproject.org/mailman/listinfo/test
