Hi，
I think due to the OVP / CVC test recording mode (only text and logging) and
test mode (vendor self-test and upload the results)
it is impossible to completely avoid fraudulent report.
A tester who familiar with OVP/CVC can easily make changes and do little
tricks that are beneficial to the SUT .
As Xu Dan said, he/she can't completely cover up the modified traces, but
this undoubtedly increases the complexity of the reviewers work.
At this stage, we can only assume that the vendor has good self-discipline in
the testing process, and obtaining the logo just a proof of product maturity.
So they will guarantee the authenticity and reliability of the test results.
However, with the increase of certification popularity and the logo
recognition,
and the number of participating vendor increases, the behavior of fraud test
results is still inevitable.
At present, OVP/CVC also declares that there will be third-party laboratories
to participate in the test. In my opinion, the third-party laboratories test is
the only way to ensure that the test results' authenticity and reliability.
Louie
------------------------------------
E-mail: yl...@biigroup.cn
Mobile: +86 13261979365
Fax: 86-10-5867-8466
Postcode：101111
Add: 2nd Floor, Building 5, No.58 Jinghai Road, BDA, Beijing, China
Website: www.biigroup.com www.cfiec.net
------------------ Original ------------------
From: "Vincent Scharf"<vincent.sch...@telekom.de>;
Date: Mon, Feb 11, 2019 05:36 PM
To: "许丹"<xuda...@huawei.com>; "tl2972"<tl2...@att.com>;
"lylavoie"<lylav...@iol.unh.edu>;
Cc: "onap-discuss"<onap-disc...@lists.onap.org>;
"victor.gao"<victor....@huawei.com>;
"compliance"<complia...@lists.lfnetworking.org>;
"opnfv-tech-discuss"<opnfv-tech-discuss@lists.opnfv.org>;
"kanagaraj.manickam"<kanagaraj.manic...@huawei.com>; "SS820F"<ss8...@att.com>;
"sw3588"<sw3...@att.com>; "mokats"<mok...@intracom-telecom.com>;
"rh173x"<rh1...@att.com>;
Subject: Re: [onap-discuss] [opnfv-tech-discuss] CVC Jonit Meeting (Feb 4th)
Agenda
Hi xudan,
I agree with you, if someone really wants to, he well always find a way to
cheat especially if the entire code is open source. Considering this, the only
reason for the entire certification topic is for operators to have a way of
pinning down the VNF provider on the ONAP compliance when making a contract.
Still the certificate would be worth nothing really. A certificate in my
understanding is something I can trust. For the OPNVF/ONAP certificate this is
not given at all. Even if I get a ONAP certified VNF I still must make my own
retest to assure, that it really is ONAP compliant. Where is the difference of
having no certificate but providing the validation software so everyone can
check themselves if a VNF is ONAP compliant?
In my opinion the only way to get a trustworthy certification process is by
moving the tests of the VNF to the LFN and automatically check the results.
Then if you’re worried about storing the VNF package due to security delete the
VNF and only keep the results, a hash and the certificate in the database.
I do not feel like this adds a lot of complexity to the process from the VNF
providers perspective. Currently he must log onto a web portal and upload
results. For the remote option he must log onto a web portal and upload the VNF
package. But for the operator it removes a lot of trouble, because the
certificate is trustworthy, and you do not have to assume that the provider
cheated and retest everything on your own.
I am pretty new to the topic, so maybe I am missing something. What are the
cons of a LFN hosted validation platform?
Best regards
Vincent
Von: xudan (N) <xuda...@huawei.com>
Gesendet: Montag, 11. Februar 2019 09:55
An: LOVETT, TREVOR J <tl2...@att.com>; Lincoln Lavoie <lylav...@iol.unh.edu>
Cc: onap-disc...@lists.onap.org; Gaoweitao (Victor, Cloudify Network OSDT)
<victor....@huawei.com>; complia...@lists.lfnetworking.org;
opnfv-tech-discuss@lists.opnfv.org; Kanagaraj Manickam
<kanagaraj.manic...@huawei.com>; STARK, STEVEN <ss8...@att.com>; WRIGHT, STEVEN
A <sw3...@att.com>; Katsaounis Molyvas Stamatios <mok...@intracom-telecom.com>;
HALLAHAN, RYAN <rh1...@att.com>; Scharf, Vincent <vincent.sch...@telekom.de>
Betreff: RE: [opnfv-tech-discuss] CVC Jonit Meeting (Feb 4th) Agenda
Hi Trevor and Lincoln,
I’d like to share some experience from OPNFV with you. OVP had the same problem
when we began to design its framework. How to ensure the uploaded results are
trusted? How to make sure the Dovetail tool hasn’t been changed locally by
testers? Then we designed a workflow with signature and checksum of all the
source code and result files. The attachment is the signature process. But at
last, OVP decide to give up all these methods which try to make it as credible
as possible. Because
1. We don’t think there can be a perfect system to avoid this. If someone
want, he/she can always find ways to get the fraudulent report and upload it.
2. All result files are related to each other. No one can make it totally
undetectable after modifying some files as the results are open for all
reviewers.
3. The testing tools are totally open source and public for everyone.
It’s easy for everyone including Providers to retest any SUT/VNF. It’s
meaningless for vendors to tamper with the results to just get the verified
badge.
Considering these, OVP abandoned all the signature and encrypt methods and make
the workflow as simple as possible.
Cheers,
Dan
From: opnfv-tech-discuss@lists.opnfv.org
[mailto:opnfv-tech-discuss@lists.opnfv.org] On Behalf Of LOVETT, TREVOR J
Sent: Friday, February 08, 2019 4:04 AM
To: Lincoln Lavoie <lylav...@iol.unh.edu>
Cc: onap-disc...@lists.onap.org; Gaoweitao (Victor, Cloudify Network OSDT)
<victor....@huawei.com>; complia...@lists.lfnetworking.org;
opnfv-tech-discuss@lists.opnfv.org; Kanagaraj Manickam
<kanagaraj.manic...@huawei.com>; STARK, STEVEN <ss8...@att.com>; WRIGHT, STEVEN
A <sw3...@att.com>; Katsaounis Molyvas Stamatios <mok...@intracom-telecom.com>;
HALLAHAN, RYAN <rh1...@att.com>; vincent.sch...@telekom.de
Subject: Re: [opnfv-tech-discuss] CVC Jonit Meeting (Feb 4th) Agenda
Thanks for describing the OPNFV flow. That’s helpful.
With regards to Option 1, I would suggest separating “where the tests are run”
from “how the results are certified”. I don’t think having the provider upload
the artifacts (Heat or CSAR) with their results necessarily constrains or
bifurcates their testing experience. They can still run these tests in the lab
and produce their results. In the infrastructure or functional testing world
recreating the testing is prohibitive. I’m sure that was a large contributor to
the certification scheme (independent review of the logs) used in OPNFV. In
this case, we’re performing rather simple, repeatable scans of static
artifacts. The provider can still develop and test externally producing their
result files. However, for this portion of the results we don’t really have
to rely on simple human inspection of forgeable text results – we can just
retest them if they provide the associated artifact. The only change would be
uploading the artifacts along with the results.
Now there might be other reasons we deem uploading the artifacts prohibitive,
but in that example the actual testing experience has not changed. We’re just
asking for one additional artifact during result submission.
To elaborate on the checksum scenario in Option 2, the checksum is a unique,
repeatable value generated from the contents of the artifact (all files). Any
change to the artifact would generate a different checksum. If a provider
submitted results and a checksum, they are stating that a specific artifact
(identifiable by the checksum) passed these tests in the result file. The OVP
portal or human reviewers can’t verify the checksum – they’re just looking at
the result text files which can always be fabricated no matter what we do.
However, the eventual consumer of the VNF can validate it. When they receive
the artifact, they can run the validations independently. The results and
checksums should match. If either is different, then either the results were
doctored or they have provided a different artifact to the consumer (different
checksum).
Happy to discuss further on the next call. I’m sure we can work out a
solution.
Thanks,
Trevor
From: Lincoln Lavoie [mailto:lylav...@iol.unh.edu]
Sent: Thursday, February 07, 2019 11:06 AM
To: LOVETT, TREVOR J <tl2...@att.com>
Cc: onap-disc...@lists.onap.org; victor....@huawei.com;
complia...@lists.lfnetworking.org; opnfv-tech-discuss@lists.opnfv.org;
Kanagaraj Manickam <kanagaraj.manic...@huawei.com>; STARK, STEVEN
<ss8...@att.com>; WRIGHT, STEVEN A <sw3...@att.com>; Katsaounis Molyvas
Stamatios <mok...@intracom-telecom.com>; HALLAHAN, RYAN <rh1...@att.com>;
vincent.sch...@telekom.de
Subject: Re: CVC Jonit Meeting (Feb 4th) Agenda
Hi Trevor,
The work flow is, user runs the dovetail tool, which produces the log output
the user can upload to the portal. This allows users to run internal testing,
debugging, etc. When they are ready to apply for certification, they upload
the results to the portal, along with submitting an application (i.e. info
about the VNF, like marketing name, etc.). That application and logs are
reviewed by the reviewers (can't be from the same company). It takes two
approvals from two different reviewers to pass that gate (i.e. 2 sets of eyes
on the results). Once approved, the admin is able to publish the listing to the
website, etc. Hopefully this clarifies things on the work flow.
For option 2, the challenge is, couldn't they just paste over the checksum
before they submit the results?
Cheers,
Lincoln
On Thu, Feb 7, 2019 at 11:57 AM LOVETT, TREVOR J <tl2...@att.com> wrote:
What was the solution in OPNFV? Can someone lay out the proposed experience and
flow as well? Based on prior conversations, I thought the proposal for a
short-term solution was a human-being from the VNF Provider/Lab to upload the
result files to the web site. If there’s an alternate proposed flow (i.e.
Dovetail directly uploading the results to the portal), then I think it would
be helpful to start getting that written down somewhere and it has bearing on
how the results are secured/certified. It’s not clear that all players are on
the same page here at this juncture.
In Option 2, I don’t fully understand your comment about a clever user
overwriting what is in the logs. I did state that the report itself could
still be fabricated so I’m not arguing that it couldn’t be. My point was that
if they provide a checksum with their report, then the result could at least be
challenged and verified. If a consumer of the VNF can’t reproduce the test
results on the files with the same checksum then the fraud can be detected.
Thanks,
Trevor
From: Lincoln Lavoie [mailto:lylav...@iol.unh.edu]
Sent: Thursday, February 07, 2019 10:22 AM
To: LOVETT, TREVOR J <tl2...@att.com>
Cc: onap-disc...@lists.onap.org; victor....@huawei.com;
complia...@lists.lfnetworking.org; opnfv-tech-discuss@lists.opnfv.org;
Kanagaraj Manickam <kanagaraj.manic...@huawei.com>; STARK, STEVEN
<ss8...@att.com>; WRIGHT, STEVEN A <sw3...@att.com>; Katsaounis Molyvas
Stamatios <mok...@intracom-telecom.com>; HALLAHAN, RYAN <rh1...@att.com>;
vincent.sch...@telekom.de
Subject: Re: CVC Jonit Meeting (Feb 4th) Agenda
Hi Trevor,
I think one concern I have with Option 1 is, it will only work for the VVP
compliance testing, as soon as we move towards more advanced testing (i.e. live
cycle testing, etc), it would cause a split path for VNF testing, where you
have to do 1/2 of your testing through a web portal and then 1/2 on some type
NFVI / MANO platform that is in your lab or another lab. I think this will just
make for a complicated future.
For option 2, this provides some protection, but a clever user could always
just overwrite what is in the logs before submitting this.
Many of these questions / points were already talked about a lot within the
OPNFV program, when that was developed and there was a desire to protect the
results as well.
Cheers,
Lincoln
On Thu, Feb 7, 2019 at 10:42 AM LOVETT, TREVOR J <tl2...@att.com> wrote:
During the call, one item that came up was how to lessen the likelihood of a
fraudulent report being submitted as part of the certification/badging process.
One suggestion was to include additional runtime, vnf-specific trace
information in the report itself making it more difficult to fabricate. The
VVP team discussed this, but we feel this would be non-trivial to implement and
not truly prevent the core issue. We would not be in favor of pursuing this
option.
If we feel that we need to protect against fraudulent reports, then we are
suggesting two alternatives that can be discussed in a future session.
1) Certification Portal Executes the validations directly using Dovetail:
a. Since we are only scanning a set of static files, and the validation
tools will be available in a container for execution, we could simply have the
portal run the scan upon uploading of the artifacts. This eliminates a
fraudulent report submission entirely, and avoids other issues such as ensuring
the proper versions of the tools are executed externally.
b. I do understand there may be sensitivity to uploading the proprietary
artifacts, but there is no need for them to be stored or shared indefinitely.
They would only need to be retained for the duration of the scan. After that
we simply need to retain the checksum of the artifact that was submitted.
c. If we simply don’t think VNF Providers would agree to that, then our
back-up option is below
2) Display Artifact Checksum Along with Badge
a. The VVP tool already computes an MD5 hash of all the file contents and
stores that in the report
b. If we retain this information and publish it on the certification site,
then the ultimate consumer of the artifact can at least verify the artifact
they receive from the vendor corresponds to the artifact submitted for
certification. We should likely do this even in option #1 for the same reason.
c. This doesn’t mean prevent the provider from submitting a fabricated
report, but it does ensure that it can be challenged and verified
after-the-fact if needed.
d. I believe the CSAR similarly has an checksum that can be used for this
purpose as well.
We can discuss the topic further in one of our upcoming joint calls.
Thanks,
Trevor
From: onap-disc...@lists.onap.org [mailto:onap-disc...@lists.onap.org] On
Behalf Of Gaoweitao(Victor)
Sent: Sunday, February 03, 2019 2:26 AM
To: complia...@lists.lfnetworking.org; onap-disc...@lists.onap.org;
opnfv-tech-discuss@lists.opnfv.org
Cc: Kanagaraj Manickam <kanagaraj.manic...@huawei.com>; Lincoln Lavoie
<lylav...@iol.unh.edu>
Subject: [onap-discuss] CVC Jonit Meeting (Feb 4th) Agenda
Hi VNFSDK/VVP/Dovetail Developers,
Here is the initial agenda for Feb 4th Joint meeting, feel free
to add your topics:
1. CVC one click deployment
2. Show marketplace capability and is it possible used for dovetail test
script execution trigger?
I will be absence due to Chinese new year and Kanagraj from VNFSDK Team is my
proxy and host the meeting.
The zoom bridge is still available during my absence:
https://zoom.us/j/346625009
The previous meeting minutes is here:
https://wiki.onap.org/display/DW/Joint+CVC+Meeting
BR
Victor
--
Lincoln Lavoie
Senior Engineer, Broadband Technologies
21 Madbury Rd., Ste. 100, Durham, NH 03824
lylav...@iol.unh.edu
https://www.iol.unh.edu
+1-603-674-2755 (m)
--
Lincoln Lavoie
Senior Engineer, Broadband Technologies
21 Madbury Rd., Ste. 100, Durham, NH 03824
lylav...@iol.unh.edu
https://www.iol.unh.edu
+1-603-674-2755 (m)
_._,_._,_
Links:
You receive all messages sent to this group.
View/Reply Online (#15431) | Reply To Group | Reply To Sender |
Mute This Topic | New Topic
Your Subscription | Contact Group Owner | Unsubscribe [yl...@biigroup.cn]
_._,_._,_-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#22782):
https://lists.opnfv.org/g/opnfv-tech-discuss/message/22782
Mute This Topic: https://lists.opnfv.org/mt/29733493/21656
Group Owner: opnfv-tech-discuss+ow...@lists.opnfv.org
Unsubscribe: https://lists.opnfv.org/g/opnfv-tech-discuss/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
