No issue with writes: 192.168.210.23:/exports/cloudstorage/primary 49T 57G 47T 1% /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020 tmpfs 6.3G 0 6.3G 0% /run/user/0 192.168.210.23:/exports/cloudstorage/secondary 49T 57G 47T 1% /var/cloudstack/mnt/161333239336.2b9f6261
[root@droid 11cd19d0-f207-3d01-880f-8d01d4b15020]# touch /var/cloudstack/mnt/161333239336.2b9f6261/file [root@droid 11cd19d0-f207-3d01-880f-8d01d4b15020]# ls -lad /var/cloudstack/mnt/161333239336.2b9f6261/file -rw-r--r-- 1 root root 0 Feb 22 17:30 /var/cloudstack/mnt/161333239336.2b9f6261/file [root@droid ~]# touch /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/file [root@droid ~]# ls -ald /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/file -rw-r--r-- 1 root root 0 Feb 22 17:31 /mnt/11cd19d0-f207-3d01-880f-8d01d4b15020/file -jeremy > On Wednesday, Feb 22, 2023 at 5:07 AM, Simon Weller <[email protected] > (mailto:[email protected])> wrote: > Jeremy, > > Any chance you have a write permission problem on your new NFS server? > Those errors indicate an underlying storage issue. > > -Si > > On Tue, Feb 21, 2023, 11:46 PM Jeremy Hansen <[email protected]> > wrote: > > > Oh and the system vm’s continue to stay in Starting state. > > > > -jeremy > > > > > > > > On Tuesday, Feb 21, 2023 at 9:44 PM, Me <[email protected]> wrote: > > The vm’s finally stopped and restarted. This is what I’m seeing in dmesg > > on the secondary storage vm: > > > > root@s-60-VM:~# dmesg | grep -i error > > [ 3.861852] blk_update_request: I/O error, dev vda, sector 6787872 op > > 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 > > [ 3.865833] blk_update_request: I/O error, dev vda, sector 6787872 op > > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > > [ 3.869553] systemd[1]: Failed to read configured hostname: > > Input/output error > > [ 4.560419] EXT4-fs (vda6): re-mounted. Opts: errors=remount-ro > > [ 4.646460] blk_update_request: I/O error, dev vda, sector 6787160 op > > 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 > > [ 4.650710] blk_update_request: I/O error, dev vda, sector 6787160 op > > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > > [ 4.975915] blk_update_request: I/O error, dev vda, sector 6787856 op > > 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 > > [ 4.980318] blk_update_request: I/O error, dev vda, sector 6787856 op > > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > > [ 5.018828] blk_update_request: I/O error, dev vda, sector 6787136 op > > 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 > > [ 5.022976] blk_update_request: I/O error, dev vda, sector 6787136 op > > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > > [ 5.026750] blk_update_request: I/O error, dev vda, sector 6787136 op > > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > > [ 5.460315] blk_update_request: I/O error, dev vda, sector 6787856 op > > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > > [ 10.415215] print_req_error: 16 callbacks suppressed > > [ 10.415219] blk_update_request: I/O error, dev vda, sector 6787864 op > > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > > [ 13.362595] blk_update_request: I/O error, dev vda, sector 6787136 op > > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > > [ 13.388990] blk_update_request: I/O error, dev vda, sector 6787136 op > > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > > [ 13.787276] blk_update_request: I/O error, dev vda, sector 6399408 op > > 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0 > > [ 13.791575] blk_update_request: I/O error, dev vda, sector 6399408 op > > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > > [ 14.632299] blk_update_request: I/O error, dev vda, sector 6787136 op > > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > > [ 14.658283] blk_update_request: I/O error, dev vda, sector 6787136 op > > 0x0:(READ) flags 0x0 phys_seg 1 prio class 0 > > > > -jeremy > > > > > > > > On Tuesday, Feb 21, 2023 at 8:57 PM, Me <[email protected]> wrote: > > The node cloudstack is claiming the system vm’s is starting on shows no > > signs of any vm’s running. virsh list is black. > > > > Thanks > > -jeremy > > > > > > > > On Tuesday, Feb 21, 2023 at 8:23 PM, Me <[email protected]> wrote: > > Also, just to note, I’m not sure how much made it in to the logs. The > > system vm’s are stuck in starting state and trying to kill through the > > interface doesn’t seem to do anything. > > > > -jeremy > > > > > > > > > > On Tuesday, Feb 21, 2023 at 8:20 PM, Me <[email protected]> wrote: > > Is there something else I can use to submit logs? Too much for pastebin. > > > > Thanks > > -jeremy > > > > > > > > On Tuesday, Feb 21, 2023 at 7:07 PM, Simon Weller <[email protected]> > > wrote: > > Can you pull some management server logs and also put the CloudStack KVM > > agent into debug mode before destroying the ssvm and share the logs? > > > > > > https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=30147350#content/view/30147350 > > > > On Tue, Feb 21, 2023, 8:33 PM Jeremy Hansen <[email protected]> > > wrote: > > > > Yes. It’s just a different partition on the same nfs server. > > > > > > > > On Tuesday, Feb 21, 2023 at 6:02 PM, Simon Weller <[email protected]> > > wrote: > > The new and old primary storage is in the same zone, correct? > > Did you also change out the secondary storage? > > > > On Tue, Feb 21, 2023, 7:59 PM Jeremy Hansen <[email protected]> > > wrote: > > > > Yes. On Kvm. I’ve been trying to destroy them from the interface and it > > just keep churning. I did a destroy with virsh, but no status changed in > > the interface. Also, the newly created ones don’t seem to bring up their > > agent and never fully start. > > > > Thanks > > > > > > > > On Tuesday, Feb 21, 2023 at 4:37 PM, Simon Weller <[email protected]> > > wrote: > > Just destroy the old system VMs and they will be recreated on available > > storage. > > > > Are you on KVM? > > > > > > > > On Tue, Feb 21, 2023, 6:14 PM Jeremy Hansen <[email protected]> > > wrote: > > > > How do I completely recreate the system vm? > > > > I was able to get the old storage in to full maintenance and deleted it, > > so maybe the system vm are still using the old storage? Is there a way to > > tell the system vm’s to use the new storage? Db change? > > > > Thanks! > > > > > > > > On Tuesday, Feb 21, 2023 at 1:36 PM, Simon Weller <[email protected]> > > wrote: > > Hey Jeremy, > > > > Is there anything in the management logs that indicate why it's not > > completing the maintenance action? > > Usually, this state is triggered by some stuck VMs that haven't migrated > > off of the primary storage. > > > > You mentioned the system VMs. Are they still on the old storage? Could > > this > > be due to some storage tags? > > > > -Si > > > > On Tue, Feb 21, 2023 at 2:35 PM Jeremy Hansen <[email protected]> > > wrote: > > > > Any ideas on this? I’m completely stuck. Can’t bring up my system vm’s > > and I can’t remove the old primary storage. > > > > -jeremy > > > > > > > > On Tuesday, Feb 21, 2023 at 2:35 AM, Me <[email protected]> wrote: > > I tried to put one of my primary storage definitions in to maintenance > > mode. Now it’s stuck in preparing for maintenance and I’m not sure how to > > remedy this situation: > > > > Cancel maintenance mode > > (NFS Primary) Resource [StoragePool:1] is unreachable: Primary storage > > with id 1 is not ready to complete migration, as the status > > is:PrepareForMaintenance > > > > Restarted manager, agents, libvirtd. My secondarystoragevm can’t start… > > > > 4.17.2.0. Using NFS for primary and secondary storage. I was attempting > > to migrate to a new volume. All volumes were moved to the new storage. I > > was simply trying to delete the old storage definition. > > > > Thanks > > -jeremy > > > > > > > > > > > > > > > >
signature.asc
Description: PGP signature
