Thanks, Joel. Based on your review and inputs the latest version is now
on the agenda of the IESG telechat this Thursday. 

Dan
 

> -----Original Message-----
> From: Joel M. Halpern [mailto:[EMAIL PROTECTED] 
> Sent: Friday, May 02, 2008 2:21 PM
> To: [email protected]
> Cc: Romascanu, Dan (Dan)
> Subject: [Fwd: Re: [Gen-art] Gen-Art IETF LC review: 
> draft-ietf-ipfix-testing-04.txt]
> 
> Since this did not make it to the spreadsheeting, I am 
> forwarding the note again.
> the -05 of the ipfix testing document addressed my concerns 
> and as far as I can tell is now ready for publication as an 
> Informational RFC.
> 
> Yours,
> Joel
> 
> -------- Original Message --------
> Subject: Re: [Gen-art] Gen-Art IETF LC review: 
> draft-ietf-ipfix-testing-04.txt
> Date: Fri, 18 Apr 2008 17:42:06 -0400
> From: Joel M. Halpern <[EMAIL PROTECTED]>
> To: Benoit Claise <[EMAIL PROTECTED]>
> CC: Romascanu, Dan (Dan) <[EMAIL PROTECTED]>, 
> [email protected], [EMAIL PROTECTED], 
> [EMAIL PROTECTED],
> [EMAIL PROTECTED]
> References: 
> <[EMAIL PROTECTED]>
> <[EMAIL PROTECTED]> <[EMAIL PROTECTED]>
> 
> Thank you.  These changes sufficiently address my concerns.
> 
> (I rather like the one case where the text reads "the tester 
> must check and if necessary improve the software so that 
> ...".  Why do I think that this is something folks have 
> tripped over before? :-)
> 
> I appreciate your efforts in addressing the issues.
> Yours,
> Joel M. Halpern
> 
> Benoit Claise wrote:
> >   Dear Joel,
> > 
> > I posted a new version of the testing draft
> > 
> >     A new version of I-D, draft-ietf-ipfix-testing-05.txt 
> has been successfuly submitted by Benoit Claise and posted to 
> the IETF repository.
> > 
> >     Filename:        draft-ietf-ipfix-testing
> >     Revision:        05
> >     Title:           Guidelines for IP Flow Information 
> eXport (IPFIX) Testing
> >     Creation_date:   2008-04-14
> >     WG ID:           ipfix
> >     Number_of_pages: 38
> > 
> >     Abstract:
> >     This document presents a list of tests for implementers 
> of IP Flow
> >     Information Export (IPFIX) compliant Exporting Processes and
> >     Collecting Processes.  This document specifies guidelines for a
> >     series of tests that can be run on the IPFIX Exporting 
> Process and
> >     Collecting Process in order to probe the conformity and 
> robustness of
> >     the IPFIX protocol implementations.  These tests cover 
> all important
> >     functions, in order to gain a level of confidence in the IPFIX
> >     implementation.  Therefore they allow the implementer to perform
> >     interoperability or plug tests with other IPFIX 
> Exporting Processes
> >     and Collecting Processes.Conventions used in this document
> > 
> > We hope to have addressed your concerns successfully. Would 
> you mind 
> > checking this latest version.
> > Note: we have addressed your point 10), as we thought we 
> didn't want 
> > to be that formal in our test description.
> > 
> > Regards, Benoit.
> > 
> > 
> >>  I have been selected as the General Area Review Team (Gen-ART) 
> >> reviewer for this draft (for background on Gen-ART, please see 
> >> http://www.alvestrand.no/ietf/gen/art/gen-art-FAQ.html ).
> >>
> >> Please resolve these comments along with any other Last 
> Call comments 
> >> you may receive.
> >>
> >> Document: Guidelines for IP Flow Information eXport (IPFIX) Testing
> >> Reviewer: Joel M. Halpern
> >> Review Date: 15-Feb-2008
> >> IETF LC End Date: 26-Feb-2008
> >> IESG Telechat date: N/A
> >>
> >> Summary: This document needs some additional work before 
> publication 
> >> as an informational RFC.
> >> I would particularly recommend considering addressing at least the 
> >> first comment below prior to RFC publication.
> >> I would also suggest that the test descriptions need some 
> >> clarification as described in the technical section below, 
> >> particularly items 5 and 6.
> >>
> >> Comments:
> >>
> >> Conceptual:
> >> 1) While the document is being published as an information 
> RFC, the 
> >> wording of the abstract and introduction make it seem that this 
> >> document is actually defining conformance to the IPFIX RFCs.  The 
> >> IETF has generally carefully steered clear of defining 
> such conformance.
> >> So, while publishing a useful test suite is probably a 
> good idea, I 
> >> strongly recommend fixing the wording of at least the abstract and 
> >> introduction to make it quite clear that these are not mandatory 
> >> tests, and that these tests do not define conformance.
> >> Related to this, please do not assert (in section 3) that passing 
> >> this test suite constitutes conformance to the IPFIX 
> architecture and 
> >> protocol.  (Among other things,test suite passage proves nothing 
> >> about architectural conformance.)
> >>
> >>
> >> Technical:
> >> 2) In the terminology section, an Observation Point is 
> defined simply 
> >> as a place where packets can be observed.  An Observation 
> Domain is a 
> >> collection of Observation points.  Then, in the middle of the 
> >> definition of an Observation domain it say "In the IPFIX 
> MEssage it 
> >> generates..." but up till now none of the things that have been 
> >> defined generate IPFIX messages.  It is possible that the 
> "it" in the 
> >> quote is supposed to be the "Metering Process" mentioned 
> in passing 
> >> earlier in the definition. But the English grammar does 
> not lead the 
> >> reader to such a conclusion. Later in that same 
> definition, it beings 
> >> to appear that an Observation Domain (which is a collection of 
> >> points, not a process or entity) is supposed to generate IPFIX 
> >> messages, since it is supposed to include a Domain ID in 
> the messages it generates.
> >> This definition for an Observation Domain needs to be reworked, to 
> >> avoid confusing the Domain with the Measurement Process which is 
> >> running in / for / on the Domain.
> >>
> >> 3) The use of capital "MUST" in section 3.1 is almost 
> certainly wrong. 
> >> Firstly, what I think that section is saying is that being able to 
> >> correctly perform the basic tests is a precondition for 
> being able to 
> >> perform further test successfully.  Thats a precondition, 
> not a "MUST".
> >> Of lesser significance, this document does not provide any 
> >> description of what it means by "MUST".  We are usually 
> careful about 
> >> how such language is used in informational RFCs.  I think 
> the meaning 
> >> would be clearer if the real intent were stated.  I 
> suspect that some 
> >> readers of this review may find my concern here pedantic.  But the 
> >> continual use of MUST in the document really, really 
> bothers me. (I 
> >> hope the next comment helps explain why it bothers me so much.)
> >>
> >> 4) Then, the test descriptions go on to keep using this language.  
> >> This is a test suite description document.  Simply state 
> how to run 
> >> the test.  There is no need for "MUST".  Section 3 should indicate 
> >> that the test descriptions describe the preconditions and 
> steps that 
> >> the tester goes through.  So section 3.1 would begin "The tester 
> >> creates one Exporting Process and one collection process, 
> configures 
> >> the Exporting Process to ..."
> >>
> >> 5) It is not clear what test steps like ~The tester 
> ensures that an 
> >> SCTP association is established.~ (Or worse, the actual text which 
> >> reads "the test MUST ensure that an SCTP association is 
> established."
> >> are supposed to do.  Is this an instruction to the tester to use 
> >> network management tools or CLI to verify a connection on both 
> >> devices?  Is it an instruction to perform additional configuration?
> >> How does the tester "ensure".  A test suite should tell a 
> tester what 
> >> steps to undertake, and what observations to perform.  "Ensure" is 
> >> not either one of those.
> >> 5a) To elaborate on this issue, in the middle of the test 
> step about 
> >> ensuring that Data Records are actually exported, we finally get a 
> >> testable instruction, to whit, use a packet sniffer and check that 
> >> the packets are coming by.
> >>
> >> 6) I believe I understand how a tester would create templates, for 
> >> the template test.  But how is the tester to create data sets.
> >> Particularly data sets with specific properties, such as 
> the padding 
> >> in section 3.2.3 and 3.2.4?  The best conclusion I can come to is 
> >> that this is a collector test, and that it assumes a 
> packet generator 
> >> which can generate IPFIX packets.  Having such a device in a test 
> >> setup makes sense.  But the test description does not say 
> "configure 
> >> a packet generator to generate an IPFIX packet with ..."  
> (There are 
> >> other ways to say this, but there needs to be some 
> description of how 
> >> testers are expected to create data sets.)
> >> 6a) Related to this, I find reading this document rather 
> odd.  I have 
> >> read many test suites for protocols and implementations of 
> protocols.
> >> They generally focus on a Device (or implementation, or 
> entity) Under 
> >> Test, and the framing around that Device.  This suite 
> appears to be 
> >> trying to test two interacting devices simultaneously.  That is 
> >> extremely difficult, and extremely confusing.  It is particularly 
> >> hard because then the tester doesn't have enough points of 
> control to 
> >> perform the tests and observe the results meaningfully.  It is 
> >> possible that this combined suite is right for this 
> problem.  But if 
> >> so, a lot of explanation of why it is done that way and how the 
> >> tester is to accomplish his goals is needed.
> >>
> >> Minor:
> >> 7) The abstract is worded as if one could not perform 
> >> interoperability testing without first running the tests in this 
> >> document.  While having run the tests in this document will 
> >> presumably increase the chances of a successful interoperability 
> >> test, they are not an inherent requirement for such testing.
> >>
> >> 8) I would probably be inclined to lighten up the 
> Motivation section 
> >> a bit.  Or even remove it.  I don't think we need to 
> explain why test 
> >> suites are useful.  If we really need a motivation 
> section, then it 
> >> should explain something about why it is particularly 
> complex to test 
> >> IPFIX implementations (if it is) and thus why the IETF feels it is 
> >> particularly useful to publish a test suite ourselves in this case.
> >>
> >> 9) The definition of Transport Session is actually the 
> definition of 
> >> various kinds of transport sessions, and how they are identified.
> >> Could the definition start with an actual definition please. (I.e. 
> >> the communication over time used to carry X between Y and Z?  Or 
> >> something.)
> >>
> >> 10) As an editorial matter, most testers I have worked 
> with strongly 
> >> prefer if every step in a test is explicitly separate and named / 
> >> numbered.  That way, they can check off each step as it 
> goes.  So the 
> >> beginning of 3.1.1 would be
> >> i) Create One Exporting Process
> >> ii) Create One Collection  Process
> >> iii) Configure the Exporting Process ...
> >>
> >> 11) It is particularly odd to see a set of Stress/Load tests that 
> >> simultaneously claim to be measuring conformance and to 
> not specify 
> >> the level of Stress / Load.  Having a description of how 
> to perform 
> >> load tests is useful.  But its relationship to the other tests is 
> >> confusing.  (This obviously is helped once we no longer claim that 
> >> this is a conformance test.)
> >>
> >>
> > 
> _______________________________________________
> Gen-art mailing list
> [email protected]
> https://www.ietf.org/mailman/listinfo/gen-art
> 
> 
_______________________________________________
Gen-art mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/gen-art

Reply via email to