Thanks for the additional info. The detailed questions I had were on how to build the domain (VM) XML configuration file outside of an OpenStack environment. While VES will support OpenStack environments and be tested with them, for non-OpenStack environments (e.g. Kubernetes) is there any dependence on the virt plugin for collectd? Our main focus going forward (beyond supporting our current OpenStack-based platform, AIC) is on cloud-native stacks that do not include OpenStack or deploy apps/VNFs in VMs.
For OpenStack environments, the info you provided helps, but I will need to test it. The issue I was having was that though I followed all the steps except for creating the domain (VM) XML configuration file, the virt plugin was not installed and thus collectd failed to find it and exited. If you're saying that just because OpenStack compute code will have been installed prior (for OpenStack hosts), the virt plugin will be installed (or the "make install" process for collectd will find the present OpenStack domain (VM) XML configuration files, and complete the job of installing the virt plugin...), then OK and I will verify that. Thanks, Bryan Sullivan | AT&T From: Mytnyk, VolodymyrX [mailto:volodymyrx.myt...@intel.com] Sent: Tuesday, November 21, 2017 7:22 AM To: SULLIVAN, BRYAN L (BRYAN L) <bryan.sulli...@research.att.com>; Tahhan, Maryam <maryam.tah...@intel.com> Cc: Mcmahon, Tony B <tony.b.mcma...@intel.com>; Power, Damien <damien.po...@intel.com>; GUPTA, ALOK <ag1...@att.com>; SINGARAJU, GOKUL <gs2...@att.com>; 'opnfv-tech-discuss@lists.opnfv.org' <opnfv-tech-discuss@lists.opnfv.org> Subject: RE: [opnfv-tech-discuss] [barometer] ves_app.py exception when collecting from multiple collectd clients Hi Bryan, To make collectd virt plugin work on a host, the libvirt library needs to be installed on the system/host (virt plugin<https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.opnfv.org_en_latest_submodules_barometer_docs_release_userguide_feature.userguide.html-23virt-2Dplugin&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=ML-JPRZQOfToJjMwlJLPlcWimAEwMA5DZGNIrk-cgy0&m=3QaIPLABF1J5bS7ZdYVUJE8lzTFK6CtloDvgmhuKd84&s=2HmUI80GTO-P-98xNvo_veQ3wEk3m2oCFz4TLHkg7bY&e=>, ` Dependencies: `). If the collectd is running in the OpenStack environment (e.g. on compute), the dependencies for libvirtd should be installed automatically during OpenStack installation. It means no other dependencies needs to be installed for collectd virt plugin. Only install collectd<https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.opnfv.org_en_latest_submodules_barometer_docs_release_userguide_collectd.ves.userguide.html-23install-2Dcollectd&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=ML-JPRZQOfToJjMwlJLPlcWimAEwMA5DZGNIrk-cgy0&m=3QaIPLABF1J5bS7ZdYVUJE8lzTFK6CtloDvgmhuKd84&s=Vcp88USCgBSGIMg5tP3gBe0OehWksYdDNv_E-_3P9G0&e=> steps needs to be performed to install collectd on a host for the VES purposes. The following commands can be used on a host to make sure that libvirt dependencies have been installed by OpenStack: $ sudo dpkg -l | grep virt ... ii libvirt-dev:amd64 2.5.0-3ubuntu5.5~cloud0 amd64 development files for the libvirt library ... If the collectd is running in non OpenStack environment, the libvirt should be installed manually (via `sudo apt-get install libvirt-dev`) or from sources (see virt plugin<https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.opnfv.org_en_latest_submodules_barometer_docs_release_userguide_feature.userguide.html-23virt-2Dplugin&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=ML-JPRZQOfToJjMwlJLPlcWimAEwMA5DZGNIrk-cgy0&m=3QaIPLABF1J5bS7ZdYVUJE8lzTFK6CtloDvgmhuKd84&s=2HmUI80GTO-P-98xNvo_veQ3wEk3m2oCFz4TLHkg7bY&e=> `Install libvirt` & ` install the dependencies: ` for more details how to build from sources). It depends on what features/extended statistics you want to receive via libvirt (see Extended statistics requirements<https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.opnfv.org_en_latest_submodules_barometer_docs_release_userguide_feature.userguide.html-23id9&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=ML-JPRZQOfToJjMwlJLPlcWimAEwMA5DZGNIrk-cgy0&m=3QaIPLABF1J5bS7ZdYVUJE8lzTFK6CtloDvgmhuKd84&s=NRfkRsR7WyxlPs9ysUo4zFJfMb0u7Wf041eSEMAA4XA&e=> section for more details). The extended statistics (if libvirt version supports them) are mapped into `additionMeasurenmets` filed of the VES message. From the VES app perspective, it shouldn't be difference what statistics are exposed by libvirt & virt plugin (e.g. what version of libvirt is installed and used). Regarding the `virsh tool`. This tool is used to create and manage VM manually via libvirt and CLI. As I understand, you are running collectd in OpenStack environment, so you don't need to use this method. The OpenStack environment should be used instead to create/manage VMs (OpenStack uses libvirt to manage VMs). If you receive collectd errors anyway during running collectd, please send me the collectd logs and config.log from the source directory (the config.log are created automatically after running `configure ...` command in the collectd source tree), so I will try to figure out the problem. Thanks and Regards, Volodymyr From: SULLIVAN, BRYAN L (BRYAN L) [mailto:bryan.sulli...@research.att.com] Sent: Tuesday, November 21, 2017 1:52 AM To: Mytnyk, VolodymyrX <volodymyrx.myt...@intel.com<mailto:volodymyrx.myt...@intel.com>>; Tahhan, Maryam <maryam.tah...@intel.com<mailto:maryam.tah...@intel.com>> Cc: Mcmahon, Tony B <tony.b.mcma...@intel.com<mailto:tony.b.mcma...@intel.com>>; Power, Damien <damien.po...@intel.com<mailto:damien.po...@intel.com>>; GUPTA, ALOK <ag1...@att.com<mailto:ag1...@att.com>>; SINGARAJU, GOKUL <gs2...@att.com<mailto:gs2...@att.com>>; 'opnfv-tech-discuss@lists.opnfv.org' <opnfv-tech-discuss@lists.opnfv.org<mailto:opnfv-tech-discuss@lists.opnfv.org>> Subject: RE: [opnfv-tech-discuss] [barometer] ves_app.py exception when collecting from multiple collectd clients OK, trying to use the node.yaml and collectd virt plugin as noted on the wiki, still having errors with collectd (collectd fails as the virt plugin is not found). I think this is probably due to some of the steps in the guide not being completed. Honestly http://docs.opnfv.org/en/latest/submodules/barometer/docs/release/userguide/feature.userguide.html#virt-plugin<https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.opnfv.org_en_latest_submodules_barometer_docs_release_userguide_feature.userguide.html-23virt-2Dplugin&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=ML-JPRZQOfToJjMwlJLPlcWimAEwMA5DZGNIrk-cgy0&m=3QaIPLABF1J5bS7ZdYVUJE8lzTFK6CtloDvgmhuKd84&s=2HmUI80GTO-P-98xNvo_veQ3wEk3m2oCFz4TLHkg7bY&e=> could be a bit clearer... it says e.g. "Create domain (VM) XML configuration file. For more information on domain XML format and examples, please see: https://libvirt.org/formatdomain.html<https://urldefense.proofpoint.com/v2/url?u=https-3A__libvirt.org_formatdomain.html&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=ML-JPRZQOfToJjMwlJLPlcWimAEwMA5DZGNIrk-cgy0&m=3QaIPLABF1J5bS7ZdYVUJE8lzTFK6CtloDvgmhuKd84&s=g5YvEljC_5mmiOUqyIrbxpX2-C6nM24Fn2IFR0rfYGY&e=>" but that's a reference to a long guide to libvirt XML files... something more specific (e.g. a working example and more clarifications as below) would be really helpful to people that aren't libvirt SMEs, e.g. * What's typical or workable for libvirt as used by OpenStack * What should be the "DOMAIN_NAME" in "virsh start DOMAIN_NAME" (maybe the name matters?) * Re "Check list of available Performance monitoring events and their settings:" what would we be checking for and what settings are important to check? E.g. which events map to measurements in the VES data model or are mappable via the node.yaml file by the ves_app (presumably we should not worry for now about those that can't be mapped). * Re "Clone and install the collectd virt plugin:" is the cloning and build/install of the collectd repo at this point redundant with the earlier build (see http://docs.opnfv.org/en/latest/submodules/barometer/docs/release/userguide/collectd.ves.userguide.html#install-collectd<https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.opnfv.org_en_latest_submodules_barometer_docs_release_userguide_collectd.ves.userguide.html-23install-2Dcollectd&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=ML-JPRZQOfToJjMwlJLPlcWimAEwMA5DZGNIrk-cgy0&m=3QaIPLABF1J5bS7ZdYVUJE8lzTFK6CtloDvgmhuKd84&s=Vcp88USCgBSGIMg5tP3gBe0OehWksYdDNv_E-_3P9G0&e=> ... at that point the goal was to "Build collectd with Kafka support" - but do we have to build twice to get the virt plugin setup?) Thanks, Bryan Sullivan | AT&T From: Mytnyk, VolodymyrX [mailto:volodymyrx.myt...@intel.com] Sent: Monday, November 20, 2017 9:05 AM To: SULLIVAN, BRYAN L (BRYAN L) <bryan.sulli...@research.att.com<mailto:bryan.sulli...@research.att.com>>; Tahhan, Maryam <maryam.tah...@intel.com<mailto:maryam.tah...@intel.com>> Cc: Mcmahon, Tony B <tony.b.mcma...@intel.com<mailto:tony.b.mcma...@intel.com>>; Power, Damien <damien.po...@intel.com<mailto:damien.po...@intel.com>>; GUPTA, ALOK <ag1...@att.com<mailto:ag1...@att.com>>; SINGARAJU, GOKUL <gs2...@att.com<mailto:gs2...@att.com>>; 'opnfv-tech-discuss@lists.opnfv.org' <opnfv-tech-discuss@lists.opnfv.org<mailto:opnfv-tech-discuss@lists.opnfv.org>> Subject: RE: [opnfv-tech-discuss] [barometer] ves_app.py exception when collecting from multiple collectd clients Hi Bryan, See my comments inline. * "How do I (as noted in the patch) get ves_app to generate messages for the guest, hypervisor, and node separately? Do I need to run 3 instances of the ves_app?" You need to run only one instance of the ves_app (patched one) somewhere where ves_app will have an access to the Kafka server (e.g. on the host). * "Should I use the yaml files in the repo or the one (merged) that you provided Friday? I assume I should use the ones in the repo." You have to use new node.yaml from gerrit review (provided on Friday, https://gerrit.opnfv.org/gerrit/#/c/47427/1/3rd_party/collectd-ves-app/ves_app/node.yaml<https://urldefense.proofpoint.com/v2/url?u=https-3A__gerrit.opnfv.org_gerrit_-23_c_47427_1_3rd-5Fparty_collectd-2Dves-2Dapp_ves-5Fapp_node.yaml&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=ML-JPRZQOfToJjMwlJLPlcWimAEwMA5DZGNIrk-cgy0&m=tAQbjXoKuI6-8IbLIJ1IFnPHe8rikjRZnuL2wqaDfmY&s=xDaI7ASpSIt1nc0vNxBrztdn8jTSJ4aq3FoVAlfb9-Y&e=> ) * Also, you have to use collectd configuration files provided on Friday. * For collectd that is running on each guest machine VM, you have to use collectd-guest.conf (please make sure IP address of Kafka server is correct) * For collectd that is running on the host machine, you have to use collectd-host.conf (please make sure IP address of Kafka server is correct) * `csv` and `logfile` plugins are enabled in the configuration files and used for debug purposes. The usage example of the `node.yaml` configuration: [cid:image004.png@01D36215.E28DDEA0] Thanks and Regards, Volodymyr From: SULLIVAN, BRYAN L (BRYAN L) [mailto:bryan.sulli...@research.att.com] Sent: Monday, November 20, 2017 6:33 PM To: Mytnyk, VolodymyrX <volodymyrx.myt...@intel.com<mailto:volodymyrx.myt...@intel.com>>; Tahhan, Maryam <maryam.tah...@intel.com<mailto:maryam.tah...@intel.com>> Cc: Mcmahon, Tony B <tony.b.mcma...@intel.com<mailto:tony.b.mcma...@intel.com>>; Power, Damien <damien.po...@intel.com<mailto:damien.po...@intel.com>>; GUPTA, ALOK <ag1...@att.com<mailto:ag1...@att.com>>; SINGARAJU, GOKUL <gs2...@att.com<mailto:gs2...@att.com>>; 'opnfv-tech-discuss@lists.opnfv.org' <opnfv-tech-discuss@lists.opnfv.org<mailto:opnfv-tech-discuss@lists.opnfv.org>> Subject: RE: [opnfv-tech-discuss] [barometer] ves_app.py exception when collecting from multiple collectd clients Bringing it back to the list, and cc'ing Alok and Gokul as some of this relates to the overall VES design. OK, ready to work on testing your patch. I can locally edit the ves-setup script to pull the patch when I clone barometer in the VES Agent (ves_app) container. Quick questions first though: * How do I (as noted in the patch) get ves_app to generate messages for the guest, hypervisor, and node separately? Do I need to run 3 instances of the ves_app? * Should I use the yaml files in the repo or the one (merged) that you provided Friday? I assume I should use the ones in the repo. Thanks, Bryan Sullivan | AT&T From: SULLIVAN, BRYAN L (BRYAN L) Sent: Monday, November 20, 2017 5:36 AM To: 'Mytnyk, VolodymyrX' <volodymyrx.myt...@intel.com<mailto:volodymyrx.myt...@intel.com>>; Tahhan, Maryam <maryam.tah...@intel.com<mailto:maryam.tah...@intel.com>> Cc: Mcmahon, Tony B <tony.b.mcma...@intel.com<mailto:tony.b.mcma...@intel.com>>; Power, Damien <damien.po...@intel.com<mailto:damien.po...@intel.com>> Subject: RE: [opnfv-tech-discuss] [barometer] ves_app.py exception when collecting from multiple collectd clients I'm using Barometer master and guest mode based upon the discussions we had (not the merged yaml). Trying to keep it simple on my end so I can get the basic single node deployment completed. That's done now as of https://gerrit.opnfv.org/gerrit/#/c/47475/<https://urldefense.proofpoint.com/v2/url?u=https-3A__gerrit.opnfv.org_gerrit_-23_c_47475_&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=ML-JPRZQOfToJjMwlJLPlcWimAEwMA5DZGNIrk-cgy0&m=tAQbjXoKuI6-8IbLIJ1IFnPHe8rikjRZnuL2wqaDfmY&s=aeV-Bg_SbIp5MRiNuJiTPctpxyqK3n9QNAUmcaAIj1I&e=> so I can start looking at your recent changes. Deploying patches from other repos gets a little complicated as I clone Barometer in the scripts, and have to edit the script to include checkout of your patch. So I usually wait until changes from other repos have merged to test. But I'll see what I can do in the meantime. Thanks, Bryan Sullivan | AT&T From: Mytnyk, VolodymyrX [mailto:volodymyrx.myt...@intel.com] Sent: Monday, November 20, 2017 5:18 AM To: SULLIVAN, BRYAN L (BRYAN L) <bryan.sulli...@research.att.com<mailto:bryan.sulli...@research.att.com>>; Tahhan, Maryam <maryam.tah...@intel.com<mailto:maryam.tah...@intel.com>> Cc: Mcmahon, Tony B <tony.b.mcma...@intel.com<mailto:tony.b.mcma...@intel.com>>; Power, Damien <damien.po...@intel.com<mailto:damien.po...@intel.com>> Subject: RE: [opnfv-tech-discuss] [barometer] ves_app.py exception when collecting from multiple collectd clients Hi Bryan, Based on the logs you provided AssertionError: Wrong SELECT condition, selected 4 metrics looks like the latest changes of VES app is not used. E.g. the https://gerrit.opnfv.org/gerrit/#/c/47427/<https://urldefense.proofpoint.com/v2/url?u=https-3A__gerrit.opnfv.org_gerrit_-23_c_47427_&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=ML-JPRZQOfToJjMwlJLPlcWimAEwMA5DZGNIrk-cgy0&m=wvfRnqnK1lxmd6iX8-opA5DLkTYOTy2WAtC-81JI7qc&s=lsPEgr9kC5dQSvNnVsMfzKNuVv4E3ZmbaqThVx6NXYc&e=> patch is not applied. 1. Are you sure that this patch is applied and node.yaml configuration is used by VES application? 2. Are you using collectd configuration files ( guests/hosts) provided by me on Friday? Thanks and Regards, Volodymyr From: SULLIVAN, BRYAN L (BRYAN L) [mailto:bryan.sulli...@research.att.com] Sent: Monday, November 20, 2017 3:09 PM To: Tahhan, Maryam <maryam.tah...@intel.com<mailto:maryam.tah...@intel.com>> Cc: Mytnyk, VolodymyrX <volodymyrx.myt...@intel.com<mailto:volodymyrx.myt...@intel.com>>; Mcmahon, Tony B <tony.b.mcma...@intel.com<mailto:tony.b.mcma...@intel.com>>; Power, Damien <damien.po...@intel.com<mailto:damien.po...@intel.com>> Subject: RE: [opnfv-tech-discuss] [barometer] ves_app.py exception when collecting from multiple collectd clients If there's a better way to report issues, e.g. thru the github mirror, let me know. I do have a number of things to followup on from the testing. Thanks, Bryan Sullivan | AT&T From: Tahhan, Maryam [mailto:maryam.tah...@intel.com] Sent: Monday, November 20, 2017 5:06 AM To: SULLIVAN, BRYAN L (BRYAN L) <bryan.sulli...@research.att.com<mailto:bryan.sulli...@research.att.com>> Cc: Mytnyk, VolodymyrX <volodymyrx.myt...@intel.com<mailto:volodymyrx.myt...@intel.com>>; Mcmahon, Tony B <tony.b.mcma...@intel.com<mailto:tony.b.mcma...@intel.com>>; Power, Damien <damien.po...@intel.com<mailto:damien.po...@intel.com>> Subject: RE: [opnfv-tech-discuss] [barometer] ves_app.py exception when collecting from multiple collectd clients Hey Bryan Thanks for the bug report. I've pinged Volodymyr on this, not sure if he follows the mailing list. BR Maryam From: opnfv-tech-discuss-boun...@lists.opnfv.org<mailto:opnfv-tech-discuss-boun...@lists.opnfv.org> [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] On Behalf Of SULLIVAN, BRYAN L (BRYAN L) Sent: Monday, November 20, 2017 4:39 AM To: 'opnfv-tech-discuss@lists.opnfv.org' <opnfv-tech-discuss@lists.opnfv.org<mailto:opnfv-tech-discuss@lists.opnfv.org>> Subject: [opnfv-tech-discuss] [barometer] ves_app.py exception when collecting from multiple collectd clients Hi Barometer team, The exception below came up when the ves_app began getting events from multiple collectd clients (one on each compute node). This occurs when there is a shared kafka server and a single ves_app instance which is getting the events from the multiple collectd clients. The exceptions started happening as soon as I activated the 2nd collectd client. The exception info "AssertionError: Wrong SELECT condition, selected 4 metrics" changed from "2 metrics" to "4 metrics" as I added the 2nd thru 4th collectd client, so I'm pretty sure the issue is related to there being more than one collectd client. Let me know if you need more info. Exception in thread Thread-130: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 1073, in run self.function(*self.args, **self.kwargs) File "/opt/ves/barometer/3rd_party/collectd-ves-app/ves_app/normalizer.py", line 561, in on_timer self.process_measurements() File "/opt/ves/barometer/3rd_party/collectd-ves-app/ves_app/normalizer.py", line 583, in process_measurements measurements = loader.get_data() File "/usr/local/lib/python2.7/dist-packages/yaml/constructor.py", line 33, in get_data return self.construct_document(self.get_node()) File "/usr/local/lib/python2.7/dist-packages/yaml/constructor.py", line 48, in construct_document for dummy in generator: File "/usr/local/lib/python2.7/dist-packages/yaml/constructor.py", line 398, in construct_yaml_map value = self.construct_mapping(node) File "/usr/local/lib/python2.7/dist-packages/yaml/constructor.py", line 208, in construct_mapping return BaseConstructor.construct_mapping(self, node, deep=deep) File "/usr/local/lib/python2.7/dist-packages/yaml/constructor.py", line 133, in construct_mapping value = self.construct_object(value_node, deep=deep) File "/usr/local/lib/python2.7/dist-packages/yaml/constructor.py", line 88, in construct_object data = constructor(self, node) File "/opt/ves/barometer/3rd_party/collectd-ves-app/ves_app/normalizer.py", line 411, in from_yaml items.append(loader.construct_mapping(item)) File "/usr/local/lib/python2.7/dist-packages/yaml/constructor.py", line 208, in construct_mapping return BaseConstructor.construct_mapping(self, node, deep=deep) File "/usr/local/lib/python2.7/dist-packages/yaml/constructor.py", line 133, in construct_mapping value = self.construct_object(value_node, deep=deep) File "/usr/local/lib/python2.7/dist-packages/yaml/constructor.py", line 88, in construct_object data = constructor(self, node) File "/opt/ves/barometer/3rd_party/collectd-ves-app/ves_app/normalizer.py", line 350, in from_yaml 'Wrong SELECT condition, selected {} metrics'.format(len(metrics)) AssertionError: Wrong SELECT condition, selected 4 metrics Thanks, Bryan Sullivan | AT&T
_______________________________________________ opnfv-tech-discuss mailing list opnfv-tech-discuss@lists.opnfv.org https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
