TDD FTW !
- Carlos Santana @csantanapr > On Jul 6, 2018, at 2:20 AM, Ben Browning <bbrow...@redhat.com> wrote: > > After several failed attempts, I have an approach that I believe will > work well enough for our needs. I pushed some in-progress code and > swagger spec fixes to a branch on a fork just so the general approach > can be shared - > https://github.com/projectodd/incubator-openwhisk/commit/5adb6be27188dcdc5b9519b57c11802a431b3e9c > > This approach uses the swagger-request-validator Java library. I'm > wiring that up to validate all requests and responses from the > WskRest* tests transparently, which you can see in > https://github.com/projectodd/incubator-openwhisk/commit/5adb6be27188dcdc5b9519b57c11802a431b3e9c#diff-5e514c6c3c27b9210d85a08d4ed3ed35 > (with some cruft and debug output that I'll clean up before submitting > a PR) > > This means that almost all of those tests are now failing due to > failed swagger spec validations. For now I'm going to concentrate on > making the spec match the reality and try to get all the existing > tests green with the validation in place and submit a PR. After that, > we can patch up any known but uncaught holes by increasing the test > coverage. > > Ben > > > >> On Thu, Jun 28, 2018 at 11:23 PM, Carlos Santana <csantan...@gmail.com> >> wrote: >> Thanks Ben for the quick update on this task >> >> -cs >> >>> On Thu, Jun 28, 2018 at 5:15 PM Ben Browning <bbrow...@redhat.com> wrote: >>> >>> After doing some preliminary poking at this, I believe we'll want to >>> use either a tool like https://github.com/google/oatts to generate a >>> test suite from our Swagger spec OR use swagger-codegen to generate a >>> Scala client from our Swagger spec and try to plug that into the >>> existing WskRest tests. >>> >>> Using the oatts tool doesn't really fit well with the test setup in >>> the existing incubator-openwhisk repo (where the API spec lives) >>> because that generates Node.js tests. >>> >>> So, I'm leaning towards the second option, which is wiring in >>> generation of a Scala client into the gradle build and having the >>> current WskRest test client use this generated Scala client for >>> testing instead of directly invoking URLs. >>> >>> However, last time I played with Scala code generated from Swagger >>> specs it wasn't that usable. So, a bit more experimentation will >>> validate whether this option is viable or if other alternatives need >>> to be considered. I already have a handful of bugs in the API spec >>> that need to be fixed but I'm waiting to fix and push those until I >>> can get some kind of testing wired up to reproduce the bugs and verify >>> the fix. >>> >>> Ben >>> >>> >>> On Thu, Jun 21, 2018 at 3:04 PM, Carlos Santana <csantan...@gmail.com> >>> wrote: >>>> Thanks Ben for looking into this, having a good API doc/spec and matching >>>> tests is very need it. >>>> >>>> +1 >>>> >>>> -cs >>>> >>>> On Thu, Jun 21, 2018 at 2:25 PM Ben Browning <bbrow...@redhat.com> >>> wrote: >>>> >>>>> Our Swagger spec >>>>> ( >>>>> >>> https://github.com/apache/incubator-openwhisk/blob/92a64c291156a2cd3d6b304babc2a193a46d0699/core/controller/src/main/resources/apiv1swagger.json >>>>> ) >>>>> is incomplete and doesn't accurately reflect the actual Controller >>>>> API. It's manually updated without a full test suite which means it's >>>>> easy for changes in the API to happen without the spec getting >>>>> updated. >>>>> >>>>> An accurate Swagger spec will not only better document the OpenWhisk >>>>> API but also allow autogenerated clients in multiple languages to >>>>> supplement or eventually replace some of the existing client >>>>> implementations we have today. It also paves way for future compatible >>>>> server implementations, whether they be rewrites of the existing >>>>> Controller or stub test harnesses to facilitate end-to-end testing on >>>>> a developer's laptop. >>>>> >>>>> As I'm already working with autogenerating code from the Swagger spec >>>>> for other purposes, I'm happy to take the lead on this effort. I'd >>>>> like to take a two-pronged approach for a test suite: >>>>> >>>>> * Generate a server stub from the spec and ensure the wsk CLI can >>>>> communicate with it. >>>>> >>>>> * Generate a client stub from the spec and ensure it can communicate >>>>> with the existing API. >>>>> >>>>> There are a lot of details to figure out from those two statements. >>>>> And, this approach won't guarantee 100% correctness of the spec. The >>>>> only way to do that would be to generate all supported clients and the >>>>> Controller API from the spec. But, this should get us started in the >>>>> right direction. >>>>> >>>>> If anyone's gone down this path before and has some wisdom to share, >>>>> please speak up! >>>>> >>>>> Thanks, >>>>> >>>>> Ben >>>>> >>>
