On Oct 2, 2014 Tom Taylor wrote:
> Summary: Ready with minor issues and nits. I did not check out the
> correctness of the code in Appendix A.
Tom: Thank you for your detailed review and your time on our draft.
Please see inline for resolutions.
Minor issues:
------------
(1) Sec. 4.10 paragraphs 3 and 4: I believe these are about setting
the initial value for the attempt rate. The first paragraph offers the
alternative of using a vendor-provided value R as a starting point or
using r=R as a starting point. Doesn't seem like two alternatives.
The second paragraph ignores all this and sets r=100 as the starting
point. But then it repeats text about using a vendor-supplied value.
The two paragraphs need cleaning up.
This is a very good comment. Here is what I propose:
OLD:
If the DUT vendor provides a value for R, the tester can use this
value. Alternatively, in cases where the DUT vendor does not provide
a value for R, or in cases where the tester wants to ascertain a
vendor provided value using local media characteristics, the
algorithm could be run by setting "r = R" and observing the value at
convergence.
The algorithm proceeds by initializing "r = 100"; "r" is the session
attempt rate. The algorithm dynamically increases and decreases "r"
as it converges to the a maximum sps value for R. The dynamic
increase and decrease rate is controlled by the weights "w" and "d",
respectively. If the DUT vendor provides a value for R, the tester
can use that value; however, because the requirements and media
characteristics are a function of the test environment, it is best
that the tester reflect these requirements during testing and allow
the algorithm to converge to R.
NEW:
If the DUT vendor provides a value for R, the tester can use this
value to initialize "r = R", and proceed with the algorithm. In
cases where the DUT vendor does not provide a value for R, or in
cases where the tester wants to ascertain a vendor provided value
using local media characteristics, it is best that the tester
reflect these requirements during testing and allow the algorithm to
converge to R. In such cases, the algorithm proceeds by initializing
"r = 100"; "r" is the session attempt rate. The algorithm
dynamically increases and decreases "r" as it converges to the a
maximum sps value for R. The dynamic increase and decrease rate is
controlled by the weights "w" and "d", respectively.
(2) The algorithm in 4.10 seems fairly unsophisticated, and amounts
to ten iterations of constrained binary search. Was application of
Newton's Method
http://www.ima.umn.edu/%7Emiller/Newton-MVT.pdf
to give you faster convergence and shorter test runs considered?
Very pertinent feedback (thanks!). Pursuant to our offline
conversations, I believe that the algorithm as specified in bmwg-sip-
bench-meth-11 does not need to be modified. In the end, the algorithm
in the draft is a version of gradient descent, or in our case, ascent.
Such algorithms may not be fast, but they do find the maxima (minima)
on a bounded number of iterations.
An important reason why the algorithm in bmwg-sip-bench-meth-11 appears
in the form that is does is for the simple reason that the tester may
not know R a-priori. Recall that R is a SIP session attempt rate,
which the DUT can process with zero errors over a defined, extended
period.
Our draft and the algorithm takes the conservative approach that the
tester may not be given the value of R by the vendor a-priori, or
the tester may independently want to approximate R to double-check
vendor claim using locally pertinent media characteristics.
The binary search method depends on knowing the value of R, since that
will become a ceiling to the process. The Newton (N-R) method also
depends on knowing the value of R, since to get N-R going, we would
start the first run at the claimed steady-state (R) capacity. I have
no doubt that we can make binary search or N-R work such that R is not
known a-priori. But by doing so I suspect that we'd be close to the
gradient descent that we have now.
Based on the simulation results for a binary search I shared with
you, I suggest adding a sentence in the revision to bench-meth as
follows:
OLD:
The pseudo-code corresponding to the description above follows, and a
simulation written in the R statistical language is provided in
Appendix A.
NEW:
The pseudo-code corresponding to the description above follows, and a
simulation written in the R statistical language is provided in
Appendix A. We note that it is possible to have other algorithms
that converge faster to R than the one shown below (e.g., an
algorithm based on a binary search), however, such algorithms may not
produce an optimal value for R, that is, they may sacrifice some
accuracy for speed. Such algorithms are for further study.
Nits/editorial comments:
-----------------------
IDNits requests that the code in Appendix A be preceded by a <CODE
BEGINS> line and terminated by a <CODE ENDS> line.
Sec. 3 para 2 last line: s/arrangements/arrangement/
Got it. Thanks!
Next para: spell out EA as presumably "Emulated Agents" on first
use?Term is from the terminology document.
Done.
Add <CODE BEGINS> and <CODE ENDS> around the pseudo-code in 4.10.
Reference [I-D.sip-bench-term] is now dated July, 2014 (version -11).
Good catch. Updated.
I will skip the <CODE BEGINS> and <CODE ENDS> nits, if it is okay with
you.
The results will show up in -12 that will be released shortly. Thanks
a lot, Tom, for your insights and feedback on this work.
Cheers,
- vijay
--
Vijay K. Gurbani, Bell Laboratories, Alcatel-Lucent
1960 Lucent Lane, Rm. 9C-533, Naperville, Illinois 60563 (USA)
Email: vkg@{bell-labs.com,acm.org} / [email protected]
Web: http://ect.bell-labs.com/who/vkg/ | Calendar: http://goo.gl/x3Ogq
_______________________________________________
Gen-art mailing list
[email protected]
https://www.ietf.org/mailman/listinfo/gen-art
