Hi Rob, Mark and All, I hear you. This will be good information to have. But, practically speaking, it's going to be difficult to collect.
Just because it's a challenge doesn't mean we shouldn't try. We can attack this thing on two fronts. A. Add the question(s) of what API calls people are using to the survey. Just need to figure out how to resolve #1 and #2 below. B. Get aggregate API call data from companies willing to share it. Is there any such effort already underway to collect this kind of data by the Foundation? There are a number of issues that stand out to me but before getting into them some quick definitions to be clear what I'm talking about. OpenStack developer = developer working on OpenStack itself OpenStack operator = operator deploying/maintaining an OpenStack cloud application developer = developer working on application being deployed on an OpenStack cloud application operator = operator deploying/maintaining an application on an OpenStack cloud users = any of OpenStack operator, application developer, application operator 1. The breadth and depth of API calls. Displaying all of the API calls possible on screen for the survey would be a challenge in itself. It would probably have to be generated and presented to the user in a hierarchical layout. I have trouble believing very many users would be wiling to drill down through the layers to check the methods they use. Maybe I'm being overly skeptical here. Presenting a subset of the API calls could work but I'm not sure how to pick the subset. This is a tricky one. Would love to hear suggestions. 2. Application developers actually taking the survey. Currently the survey is targeted towards OpenStack operators. Not to say we can't switch the focus but it'll be a bit more difficult to get application developers/operators in front of it just because they're a bit more removed from OpenStack. We could ask the OpenStack operators to ask their application developers/operators to take it and give them an email template for them to send it out or something along those lines. Perhaps it's time to open the survey with a "What kind of user are you?" type question. 3. Application developers only knowing their single system The feedback from an app dev is useful but they really only know about their own system. What we really need is an aggregate of all of the API calls being made. 4. The difference between application developers and application operators They are going to have different needs and make different API calls. An app dev might rely heavily on Marconi and never call Nova and vice versa for an app operator. Again we really need aggregate numbers. To get real aggregate numbers the best thing to do is to go the logs. To me this appears to be a daunting challenge. Hurdles include: getting companies to agree to allow this and share their aggregates, logging every API call (is this already done?), logging configuration, aggregation (the logs could be spread over many systems or, hopefully, centralized), aggregation from archived logs, etc. This is where B from above comes in. Let me know what you think. Thanks, Everett On Jan 20, 2014, at 12:15 AM, <rob_hirschf...@dell.com<mailto:rob_hirschf...@dell.com>> <rob_hirschf...@dell.com<mailto:rob_hirschf...@dell.com>> wrote: Mark & Everett, I agree that there’s good potential alignment w/ the DefCore data needs. It would be very interesting to know which API calls are required by different client libraries. That would help identify classes of tests for review. We’ve already captured that as a desired criteria for weighting but I don’t know how we’re doing to determine that on a test-by-test basis! Rob From: Mark Collier [mailto:m...@collierclan.net] Sent: Friday, January 17, 2014 4:28 PM To: Everett Toews Cc: openstack@lists.openstack.org<mailto:openstack@lists.openstack.org>; user-commit...@lists.openstack.org<mailto:user-commit...@lists.openstack.org> Subject: Re: [Openstack] Formulate application developer oriented questions for the user survey Importance: Low Thanks for bringing up the opportunity to expand the survey to gather feedback from end-users (of the API...) Everett. I think this is a great idea, and I was also thinking that as we make progress on the interop efforts with things like DefCore[1][2][3] (and RefStack) that Rob Hirshfeld & others are leading, it's critical that we get input from this class of "user". For example, it would be ideal IMHO to identify which APIs (and underlying functions) are most valued, to inform the list of tests included and ultimately the ones that "must pass". Today there is an optional component in the survey for "deployment" which obviously fits the operator class of "user", so perhaps there is a similar path with more detailed question for the devs targeting the APIs (i.e. deploying apps on openstack clouds, for lack of a better phrase) that includes some kind of feedback mechanism for individual APIs (don't care /nice to have/ must have) as well as related issues like api discoverability. [1] http://robhirschfeld.com/category/clouds/openstack/defcore/ [2] https://wiki.openstack.org/wiki/Governance/DefCoreCommittee [3] http://lists.openstack.org/cgi-bin/mailman/listinfo/defcore-committee On Fri, Jan 17, 2014 at 2:09 PM, Everett Toews <everett.to...@rackspace.com<mailto:everett.to...@rackspace.com>> wrote: On Jan 17, 2014, at 5:45 AM, Gregor von Laszewski wrote: > Everett: > > you may want to add a question such as > > * “Why did you use this library?” That's a good point. One of the things that concerns me about how the questions are worded right now is that they are very reactionary. That's great for finding out the current state of thing but does nothing to inform us on where users want to go in the future and how we can help them get there. They're more "What are you using right now?" not "What will you need in the future?" Asking why will also help reveal requirements. Personally I need to think on this a bit more and take another crack at the questions later. > This may give an additional insight and possibly motivation for further > actions/development. Some examples > > a) in Python the use of libcloud via the EC2 is popular. Why?: compatibility > > b) in the cloudmesh project we developed our own compatibility library that > makes use of the native openstack protocol instead of using lib cloud/EC2 to > access openstack. Why: (1) libcloud/EC2 has limited functionality, (2) > debugging of production clouds with native protocols (starting thousands of > vms), (3) easier integration into user interfaces while leveraging JSON. (4) > Together this allows us to have an API that accesses and manages VMS on AWS, > Azure, and Openstack the same way but uses in case of Openstack the native > protocol instead of lib cloud/EC2. Just out of curiosity, why didn't you contribute to the libcloud project to fill in the missing functionality rather than start your own? Thanks, Everett _______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org<mailto:openstack@lists.openstack.org> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
