I've submitted a PR for the Swagger spec changes and validations at https://github.com/apache/incubator-openwhisk/pull/3878
For this first iteration, I just focused on making the Swagger spec match the behavior of our existing tests. As Rodric pointed out, that may not exactly match the REST interface, but it at least gets us closer. As a future second iteration, I'd like to investigate generating and maintaining a new Java OpenWhisk client from the spec (see https://github.com/apache/incubator-openwhisk/issues/2466). Using the spec to generate and test a functioning Java client should help find and fix more spec issues and pave the way for potentially moving other hand-generated client libraries over to being at least partially generated from the API spec. Ben On Fri, Jul 6, 2018 at 12:55 AM, Rodric Rabbah <rod...@gmail.com> wrote: > Ben, you might want to take note of this PR https://github.com/apache/ > incubator-openwhisk/pull/3840 > which removes a number of tests suites (redundant with unit tests). Also > I've found that the WskRestOperations implementation in some places fixes > behaviors to match the CLI instead of strictly implementing the > narrower/strict REST interface. This may mean you have to edit tests (or > remove some that are only valid for clients). > > -r > > On Thu, Jul 5, 2018 at 9:20 PM, Ben Browning <bbrow...@redhat.com> wrote: > >> After several failed attempts, I have an approach that I believe will >> work well enough for our needs. I pushed some in-progress code and >> swagger spec fixes to a branch on a fork just so the general approach >> can be shared - >> https://github.com/projectodd/incubator-openwhisk/commit/5ad >> b6be27188dcdc5b9519b57c11802a431b3e9c >> >> This approach uses the swagger-request-validator Java library. I'm >> wiring that up to validate all requests and responses from the >> WskRest* tests transparently, which you can see in >> https://github.com/projectodd/incubator-openwhisk/commit/5ad >> b6be27188dcdc5b9519b57c11802a431b3e9c#diff-5e514c6c3c27b9210 >> d85a08d4ed3ed35 >> (with some cruft and debug output that I'll clean up before submitting >> a PR) >> >> This means that almost all of those tests are now failing due to >> failed swagger spec validations. For now I'm going to concentrate on >> making the spec match the reality and try to get all the existing >> tests green with the validation in place and submit a PR. After that, >> we can patch up any known but uncaught holes by increasing the test >> coverage. >> >> Ben >> >> >> >> On Thu, Jun 28, 2018 at 11:23 PM, Carlos Santana <csantan...@gmail.com> >> wrote: >> > Thanks Ben for the quick update on this task >> > >> > -cs >> > >> > On Thu, Jun 28, 2018 at 5:15 PM Ben Browning <bbrow...@redhat.com> >> wrote: >> > >> >> After doing some preliminary poking at this, I believe we'll want to >> >> use either a tool like https://github.com/google/oatts to generate a >> >> test suite from our Swagger spec OR use swagger-codegen to generate a >> >> Scala client from our Swagger spec and try to plug that into the >> >> existing WskRest tests. >> >> >> >> Using the oatts tool doesn't really fit well with the test setup in >> >> the existing incubator-openwhisk repo (where the API spec lives) >> >> because that generates Node.js tests. >> >> >> >> So, I'm leaning towards the second option, which is wiring in >> >> generation of a Scala client into the gradle build and having the >> >> current WskRest test client use this generated Scala client for >> >> testing instead of directly invoking URLs. >> >> >> >> However, last time I played with Scala code generated from Swagger >> >> specs it wasn't that usable. So, a bit more experimentation will >> >> validate whether this option is viable or if other alternatives need >> >> to be considered. I already have a handful of bugs in the API spec >> >> that need to be fixed but I'm waiting to fix and push those until I >> >> can get some kind of testing wired up to reproduce the bugs and verify >> >> the fix. >> >> >> >> Ben >> >> >> >> >> >> On Thu, Jun 21, 2018 at 3:04 PM, Carlos Santana <csantan...@gmail.com> >> >> wrote: >> >> > Thanks Ben for looking into this, having a good API doc/spec and >> matching >> >> > tests is very need it. >> >> > >> >> > +1 >> >> > >> >> > -cs >> >> > >> >> > On Thu, Jun 21, 2018 at 2:25 PM Ben Browning <bbrow...@redhat.com> >> >> wrote: >> >> > >> >> >> Our Swagger spec >> >> >> ( >> >> >> >> >> https://github.com/apache/incubator-openwhisk/blob/92a64c291 >> 156a2cd3d6b304babc2a193a46d0699/core/controller/src/main/ >> resources/apiv1swagger.json >> >> >> ) >> >> >> is incomplete and doesn't accurately reflect the actual Controller >> >> >> API. It's manually updated without a full test suite which means it's >> >> >> easy for changes in the API to happen without the spec getting >> >> >> updated. >> >> >> >> >> >> An accurate Swagger spec will not only better document the OpenWhisk >> >> >> API but also allow autogenerated clients in multiple languages to >> >> >> supplement or eventually replace some of the existing client >> >> >> implementations we have today. It also paves way for future >> compatible >> >> >> server implementations, whether they be rewrites of the existing >> >> >> Controller or stub test harnesses to facilitate end-to-end testing on >> >> >> a developer's laptop. >> >> >> >> >> >> As I'm already working with autogenerating code from the Swagger spec >> >> >> for other purposes, I'm happy to take the lead on this effort. I'd >> >> >> like to take a two-pronged approach for a test suite: >> >> >> >> >> >> * Generate a server stub from the spec and ensure the wsk CLI can >> >> >> communicate with it. >> >> >> >> >> >> * Generate a client stub from the spec and ensure it can communicate >> >> >> with the existing API. >> >> >> >> >> >> There are a lot of details to figure out from those two statements. >> >> >> And, this approach won't guarantee 100% correctness of the spec. The >> >> >> only way to do that would be to generate all supported clients and >> the >> >> >> Controller API from the spec. But, this should get us started in the >> >> >> right direction. >> >> >> >> >> >> If anyone's gone down this path before and has some wisdom to share, >> >> >> please speak up! >> >> >> >> >> >> Thanks, >> >> >> >> >> >> Ben >> >> >> >> >> >>
