On Dec 3, 2013, at 12:18 AM, Joe Gordon <joe.gord...@gmail.com> wrote:
> > > > On Sun, Dec 1, 2013 at 7:04 PM, John Dickinson <m...@not.mn> wrote: > Just to add to the story, Swift uses "X-Trans-Id" and generates it in the > outer-most "catch_errors" middleware. > > Swift's catch errors middleware is responsible for ensuring that the > transaction id exists on each request, and that all errors previously > uncaught, anywhere in the pipeline, are caught and logged. If there is not a > common way to do this, yet, I submit it as a great template for solving this > problem. It's simple, scalable, and well-tested (ie tests and running in prod > for years). > > https://github.com/openstack/swift/blob/master/swift/common/middleware/catch_errors.py > > Leaving aside error handling and only focusing on the transaction id (or > request id) generation, since OpenStack services are exposed to untrusted > clients, how would you propose communicating the appropriate transaction id > to a different service? I can see great benefit to having a glance > transaction ID carry through to Swift requests (and so on), but how should > the transaction id be communicated? It's not sensitive info, but I can > imagine a pretty big problem when trying to track down errors if a client > application decides to set eg the X-Set-Transaction-Id header on every > request to the same thing. > > -1 to cross service request IDs, for the reasons John mentions above. > > > Thanks for bringing this up, and I'd welcome a patch in Swift that would use > a common library to generate the transaction id, if it were installed. I can > see that there would be huge advantage to operators to trace requests through > multiple systems. > > Another option would be for each system that calls an another OpenStack > system to expect and log the transaction ID for the request that was given. > This would be looser coupling and be more forgiving for a heterogeneous > cluster. Eg when Glance makes a call to Swift, Glance cloud log the > transaction id that Swift used (from the Swift response). Likewise, when > Swift makes a call to Keystone, Swift could log the Keystone transaction id. > This wouldn't result in a single transaction id across all systems, but it > would provide markers so an admin could trace the request. > > There was a session on this at the summit, and although the notes are a > little scarce this was the conclusion we came up with. Every time a cross > service call is made, we will log and send a notification for ceilometer to > consume, with the request-ids of both request ids. One of the benefits of > this approach is that we can easily generate a tree of all the API calls that > are made (and clearly show when multiple calls are made to the same service), > something that just a cross service request id would have trouble with. Is wise to trust anything a client provides to ensure traceability? If a user receives a request id back from Nova, then submits that request id in an unrelated request to Neutron, the traceability would be effectively corrupted. If the consensus is that we don't want to securely deliver request ids for inter-service calls, how about requiring a service to log its request id along with the request id returned from a call to another service to achieve the a similar result? The catch is that every call point (or client instantiation?) would have to be modified to pass the request id instead of just logging at one place in each service. Is that a cost worth paying? m. > > https://etherpad.openstack.org/p/icehouse-summit-qa-gate-debugability > > > With that in mind I think having a standard x-openstack-request-id makes > things a little more uniform, and means that adding new services doesn't > require new logic to handle new request ids. _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
