Jeremy, Any chance you have a write permission problem on your new NFS server? Those errors indicate an underlying storage issue.
-Si On Tue, Feb 21, 2023, 11:46 PM Jeremy Hansen <[email protected]> wrote: > Oh and the system vm’s continue to stay in Starting state. > > -jeremy > > > > On Tuesday, Feb 21, 2023 at 9:44 PM, Me <[email protected]> wrote: > The vm’s finally stopped and restarted. This is what I’m seeing in dmesg > on the secondary storage vm: > > root@s-60-VM:~# dmesg | grep -i error > [ 3.861852] blk_update_request: I/O error, dev vda, sector 6787872 op > 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 > [ 3.865833] blk_update_request: I/O error, dev vda, sector 6787872 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [ 3.869553] systemd[1]: Failed to read configured hostname: > Input/output error > [ 4.560419] EXT4-fs (vda6): re-mounted. Opts: errors=remount-ro > [ 4.646460] blk_update_request: I/O error, dev vda, sector 6787160 op > 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 > [ 4.650710] blk_update_request: I/O error, dev vda, sector 6787160 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [ 4.975915] blk_update_request: I/O error, dev vda, sector 6787856 op > 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 > [ 4.980318] blk_update_request: I/O error, dev vda, sector 6787856 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [ 5.018828] blk_update_request: I/O error, dev vda, sector 6787136 op > 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 > [ 5.022976] blk_update_request: I/O error, dev vda, sector 6787136 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [ 5.026750] blk_update_request: I/O error, dev vda, sector 6787136 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [ 5.460315] blk_update_request: I/O error, dev vda, sector 6787856 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [ 10.415215] print_req_error: 16 callbacks suppressed > [ 10.415219] blk_update_request: I/O error, dev vda, sector 6787864 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [ 13.362595] blk_update_request: I/O error, dev vda, sector 6787136 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [ 13.388990] blk_update_request: I/O error, dev vda, sector 6787136 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [ 13.787276] blk_update_request: I/O error, dev vda, sector 6399408 op > 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 > [ 13.791575] blk_update_request: I/O error, dev vda, sector 6399408 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [ 14.632299] blk_update_request: I/O error, dev vda, sector 6787136 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > [ 14.658283] blk_update_request: I/O error, dev vda, sector 6787136 op > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > > -jeremy > > > > On Tuesday, Feb 21, 2023 at 8:57 PM, Me <[email protected]> wrote: > The node cloudstack is claiming the system vm’s is starting on shows no > signs of any vm’s running. virsh list is black. > > Thanks > -jeremy > > > > On Tuesday, Feb 21, 2023 at 8:23 PM, Me <[email protected]> wrote: > Also, just to note, I’m not sure how much made it in to the logs. The > system vm’s are stuck in starting state and trying to kill through the > interface doesn’t seem to do anything. > > -jeremy > > > > > On Tuesday, Feb 21, 2023 at 8:20 PM, Me <[email protected]> wrote: > Is there something else I can use to submit logs? Too much for pastebin. > > Thanks > -jeremy > > > > On Tuesday, Feb 21, 2023 at 7:07 PM, Simon Weller <[email protected]> > wrote: > Can you pull some management server logs and also put the CloudStack KVM > agent into debug mode before destroying the ssvm and share the logs? > > > https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=30147350#content/view/30147350 > > On Tue, Feb 21, 2023, 8:33 PM Jeremy Hansen <[email protected]> > wrote: > > Yes. It’s just a different partition on the same nfs server. > > > > On Tuesday, Feb 21, 2023 at 6:02 PM, Simon Weller <[email protected]> > wrote: > The new and old primary storage is in the same zone, correct? > Did you also change out the secondary storage? > > On Tue, Feb 21, 2023, 7:59 PM Jeremy Hansen <[email protected]> > wrote: > > Yes. On Kvm. I’ve been trying to destroy them from the interface and it > just keep churning. I did a destroy with virsh, but no status changed in > the interface. Also, the newly created ones don’t seem to bring up their > agent and never fully start. > > Thanks > > > > On Tuesday, Feb 21, 2023 at 4:37 PM, Simon Weller <[email protected]> > wrote: > Just destroy the old system VMs and they will be recreated on available > storage. > > Are you on KVM? > > > > On Tue, Feb 21, 2023, 6:14 PM Jeremy Hansen <[email protected]> > wrote: > > How do I completely recreate the system vm? > > I was able to get the old storage in to full maintenance and deleted it, > so maybe the system vm are still using the old storage? Is there a way to > tell the system vm’s to use the new storage? Db change? > > Thanks! > > > > On Tuesday, Feb 21, 2023 at 1:36 PM, Simon Weller <[email protected]> > wrote: > Hey Jeremy, > > Is there anything in the management logs that indicate why it's not > completing the maintenance action? > Usually, this state is triggered by some stuck VMs that haven't migrated > off of the primary storage. > > You mentioned the system VMs. Are they still on the old storage? Could > this > be due to some storage tags? > > -Si > > On Tue, Feb 21, 2023 at 2:35 PM Jeremy Hansen <[email protected]> > wrote: > > Any ideas on this? I’m completely stuck. Can’t bring up my system vm’s > and I can’t remove the old primary storage. > > -jeremy > > > > On Tuesday, Feb 21, 2023 at 2:35 AM, Me <[email protected]> wrote: > I tried to put one of my primary storage definitions in to maintenance > mode. Now it’s stuck in preparing for maintenance and I’m not sure how to > remedy this situation: > > Cancel maintenance mode > (NFS Primary) Resource [StoragePool:1] is unreachable: Primary storage > with id 1 is not ready to complete migration, as the status > is:PrepareForMaintenance > > Restarted manager, agents, libvirtd. My secondarystoragevm can’t start… > > 4.17.2.0. Using NFS for primary and secondary storage. I was attempting > to migrate to a new volume. All volumes were moved to the new storage. I > was simply trying to delete the old storage definition. > > Thanks > -jeremy > > > > > > > >
