Hello William, I have not seen any warning in our nova-scheduler logs. We have verbose enabled but not debug mode, we have provisioned far more on cinder volumes than we have available local disk space on the hypervisors and we have not yet had any issues except for the fact that this is broken and a fix is being worked in.
Best regards Tobias On 08/19/2016 03:39 PM, William Josefsson wrote: > Hi tobias > > yes, I tested today as well. With DiskFilter removed from the nova > default scheduling, one can still do a nova live-migration, no issues > so far experienced. > > I think the nova-scheduler.log may throw a warning if you provision a > number of flavors and the total image disk size for all instances is > greater than / on the compute node, the the there may be a warning at > least with the DiskFilter enabled, have you seen any warnings or > errors in logs with DiskFilter removed related to scheduling? thx will > > > > On Thu, Aug 18, 2016 at 2:40 PM, Tobias Urdin <tobias.ur...@crystone.com> > wrote: >> Hello William, >> >> We are booting our instances from Cinder volumes over ISCSI with >> multipath and can perform live migration just fine. >> >> The things we have changed: >> >> nova.conf on compute nodes >> >> [libvirt] >> virt_type=kvm >> inject_password=true >> inject_key=true >> inject_partition=-1 >> live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE,VIR_MIGRATE_PEER2PEER,VIR_MIGRATE_LIVE >> iscsi_use_multipath = True >> >> The live_migration_flag is deprecated and will be removed in next >> release I think, but we are running Liberty stable and haven't upgraded yet. >> >> Also we had issues with live migration and multipath volumes turning >> into a failed state. Changing the failback and retry settings for >> multipathd solved it (see https://bugs.launchpad.net/nova/+bug/1525802) >> >> multipath.conf on compute nodes >> >> defaults { >> failback immediate >> no_path_retry 0 >> } >> >> Best regards >> >> Tobias >> >> >> On 08/15/2016 05:15 PM, William Josefsson wrote: >>> yes thx tobias, did you try live-migration after this change? I will >>> try on my end too and share whether it worked with the DiskFilter >>> disabled or not. thx will >>> >>> On Mon, Aug 15, 2016 at 4:22 PM, Tobias Urdin <tobias.ur...@crystone.com> >>> wrote: >>>> Hello, >>>> >>>> We are having this issue in production and I have since pushed for a fix >>>> for this. >>>> >>>> We solved it by removing the DiskFilter filter, ignore any RT data from >>>> nova and seen in horizon and keep close monitoring on our block storage >>>> usage. >>>> >>>> nova.conf: >>>> >>>> #scheduler_default_filters = RetryFilter, AvailabilityZoneFilter, >>>> RamFilter, DiskFilter, ComputeFilter, ComputeCapabilitiesFilter, >>>> ImagePropertiesFilter, ServerGroupAntiAffinityFilter, >>>> ServerGroupAffinityFilter >>>> scheduler_default_filters = RetryFilter, AvailabilityZoneFilter, >>>> RamFilter, ComputeFilter, ComputeCapabilitiesFilter, >>>> ImagePropertiesFilter, ServerGroupAntiAffinityFilter, >>>> ServerGroupAffinityFilter >>>> >>>> Default options is the one commented out, the other one has DiskFilter >>>> removed. >>>> >>>> We haven't had any issues after removing the DiskFilter. >>>> >>>> >>>> Best regards >>>> >>>> >>>> On 08/15/2016 10:17 AM, kostiantyn.volenbovs...@swisscom.com wrote: >>>>> Hi, >>>>> >>>>> Just to add that this bug was recently brought up in openstack-dev [1] >>>>> >>>>> it sounds that you are waiting for fix via [2] or, probably more likely >>>>> [3] >>>>> Disabling DiskFilter sounds like OK workaround in case your VMs are never >>>>> booted from image in your OpenStack Region(s). >>>>> But if I understand the fault - the migrate/resize/live >>>>> migrate/rebuild/evacuate will not work because >>>>> . assuming that on target Compute Host you don't have local disk - and >>>>> your flavor has more than 0 GB for root and/or ephemeral disk >>>>> -it would fail. >>>>> >>>>> So I would say that setting your root disk space to '0' in the flavor >>>>> might be a better workaround >>>>> >>>>> BR, >>>>> Konstantin >>>>> [1] >>>>> http://lists.openstack.org/pipermail/openstack-dev/2016-August/101566.html >>>>> [2] https://review.openstack.org/#/c/200870/ >>>>> [3] https://review.openstack.org/#/c/355091/ >>>>> >>>>> >>>>>> -----Original Message----- >>>>>> From: William Josefsson [mailto:william.josef...@gmail.com] >>>>>> Sent: Sunday, August 14, 2016 5:47 PM >>>>>> To: openstack-operators@lists.openstack.org >>>>>> Subject: [Openstack-operators] DiskFilter Disabled - No valid host was >>>>>> found >>>>>> >>>>>> Hello, I got Error, no valid host was found while trying to provision >>>>>> and I could in >>>>>> nova-scheduler.log see that the DiskFilter failed to find a compute >>>>>> host. My OS is >>>>>> CentOS7/ running Liberty. >>>>>> >>>>>> If I'm not mistaken, it is a bug that DiskFilter checks space available >>>>>> on compute- >>>>>> host / partition even when I'm using iSCSI and RBD backends. The / is no >>>>>> way >>>>>> near full. >>>>>> https://bugs.launchpad.net/nova/+bug/1358566 >>>>>> >>>>>> Can anyone confirm whether it is fine to disable DiskFilter if I'm using >>>>>> iSCSI and >>>>>> RBD backend or would anything else brake? thx will >>>>>> >>>>>> _______________________________________________ >>>>>> OpenStack-operators mailing list >>>>>> OpenStack-operators@lists.openstack.org >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >>>>> _______________________________________________ >>>>> OpenStack-operators mailing list >>>>> OpenStack-operators@lists.openstack.org >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators >>>>> _______________________________________________ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
