On our non-trivial application, we have broken our testing into the following sets:
* Unit Tests -- written by devs, run as part of our integration builder and when doing dev * Integration Tests -- automated, hitting our external APIs, written in clojure, maintained by the devs mostly, run as part of integration builder, and occasionally during dev * System Test -- Performed by QA team, manual and automated (using Ruby and pythong) against our external APIs, performed as final phase of our 2 week release cycle Domain knowledge (malware analysis) is a bigger limiter than clojure experience when it comes to our testing. Our QA team doesn't know clojure at all. They use the same public facing APIs for their tests that we expose to the customers. As far as infrastructure management, the same mechanism we use to deploy a production instance is used to deploy a staging and testing instance. There is nothing clojure specific about this, in fact it's Arch linux packages, systemd, and our own hand-rolled "tachikoma" orchestration tool. I would not try and pull in everything into your system and have deployment/configuration managed by clojure. I would not have testers writing unit tests. I would not have coders writing System Tests. Integration tests, depending on what interface they are targeting, can be done in any language. Most of ours are in clojure because we wrote an HTTP client early on, clj-mook, and didn't have a dedicatd QA team. Since then, we have integration tests implemented in a few different languages. In short, our system seems quite similiar to yours, but we are not trying to unify our test stack, or take over deploy/config in clojure (even tho we're at the core of the whole system). We leave that to the tools our Ops and QA team selects, and I don't see sufficient win in unifying all of them under clojure to justify that. On Mon, Oct 27, 2014 at 12:21 AM, David Mitchell <monch1...@gmail.com> wrote: > Hi group, > > Apologies for the somewhat cryptic subject line - I'll try to explain... > Apologies also for the length of the post, but I'm sure others will hit the > same problem if they haven't already done so, and hopefully this discussion > will help them find a way out of a sticky situation. > > We've got a (notionally agile) Clojure app under heavy development. The > project itself follows the Agile Manifesto to a degree, but is constrained > in having to interface with other applications that are following a > waterfall process. Yep, it's awkward, but that's not what I'm asking about. > > Simplifying it as much as possible, we started with a pre-existing, > somewhat clunky, Java app, then extended the server side extensively using > Clojure, and added a web client. There's loads of (non-Clojure) supporting > infrastructure - database cluster, queue servers, identity management, > etc. At any point, we've got multiple streams of Clojure development going > on, hitting different parts of the app. The web client development is > "traditional" in that it's not using ClojureScript, and probably won't in > the foreseeable future. As mentioned above, a key point is that the app > has a significant requirement to interface to legacy systems - other Java > apps, SAP, Oracle identity management stack and so on. > > From a testing perspective, for this app we've got unit tests written in > Clojure/midje which are maintained by the app developers (as you'd > expect). These work well and midje is a good fit for the app. However, > given all the various infrastructure requirements of the app, it's hard to > see how we can use midje to go all the way up the testing stack (unit -> > system -> integration -> pre-production -> production). > > From the web client perspective, we've got UI automation tests written > using Ruby/Capybara, a toolset which I suspect was chosen based on the > existing skillset of the pool of testers. Again this works well for us. > > The problem is with the "middle ground" between the two extremes of unit > and UI testing - our glaring problem at present is with integration > testing, but there's also a smaller problem with system testing. We're > struggling to find an approach that works here, given the skillsets we have > on hand - fundamentally, we've got a (small) pool of developers who know > Clojure, a (small) pool of testers who know Ruby, and a larger pool of > testers who do primarily non-automated testing. > > In an ideal world, we'd probably use Clojure for all automated testing. > It seems relatively straightforward to use Stuart Sierra's component > library (https://github.com/stuartsierra/component) to mock out > infrastructure components such as databases, queues, email servers etc., > and doing so would let us address our system-level testing. > > On the integration front, we could conceivably also leverage the same > component library to manage the state of all the various infrastructure > components that the app depends on, and thus ensure that we had a suitably > production-like environment for integration testing. This would be a > non-trivial piece of work. > > Our big problem really boils down to just not having enough skilled > Clojure people available to the project. You could point to any of the > following areas that are probably common to any non-trivial Clojure > application: either we don't have enough Clojure developers to address the > various requirements of system and integration testing, or our techops guys > don't have the necessary skills to expose a Clojure/component interface to > the various test/development environments, or our testers don't know > Clojure and not willing to take the word of developers that their Clojure > tests are both fit for purpose and sufficient from a risk management > perspective. > > Obvious options, none of which seem great: > > - hire more Clojure people (expensive, as they're still pretty rare) > and put them to work in testing & techops. We've tried turning some of our > Clojure devs into techops already, but strangely devs who've taken the time > and had the initiative to learn Clojure don't like doing techops work. > What a surprise ;-> I suspect the same would apply if we tried turning > them into testers > - retrain our testers so they can write automated tests in Clojure. > That would be quite a stretch for our testers, and I'd suggest it would be > the same for most testers out there (otherwise they'd probably be working > as developers). Another factor is career development: once testers start > to move on from this project, what is the chance that "Clojure" will be a > useful thing to have on their CVs? > - retrain techops people so they can wrap up and expose their > infrastructure using Clojure components, making it available in such a way > that it would better support integration and systems testing. Same > problems here as with retraining testers to use Clojure > - enforce the use of :pre and :post conditions in all our Clojure > code, to bring in a "design by contract" approach and try to reduce the > "surprises" that occur during integration of different streams of work. > Aside from being a sizeable piece of work to do this, we're stuck with a > strong reliance on the pre-existing Java app and I can't see a way of > reducing the integration risk of this element > - use something other than Clojure (e.g. Ruby) for systems and > integration testing, so we could leverage the existing skillsets of our > test workforce. This is probably conceivable if we made the effort to > expose much of the functionality of the app using something like REST APIs, > but it would require a significant investment in time and would have no > likely future benefit beyond making it easier to test. I realise that's a > desirable aim in itself, but it's a hard sell to the people who pay the > bills! > > > I'll go out on a limb and suggest that, as of late 2014, probably any > non-trivial Clojure project doesn't have enough skilled Clojure people on > board to cover all the testing and operational requirements for the > project. How then are you addressing the non-development requirements of > your project that require Clojure expertise - especially testing and devops? > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
