Basic Zone - Yes router has been actually started/created on KVM side - not created, not started. Thats the main problem, I guess
agent.log https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq- management log https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A >> Can you confirm your zone/pod/cluster/hosts are all in Enabled state, i.e. YES, all green >> Can you connect your both KVM hosts can access/mount both Primary and Secondary Storage YES. Double checked >>>Can you also explain your infrastructure - you said you have two hosts only, where does CloudStack management run? 2 hosts: host1: 192.168.1.14 host2: 192.168.1.5 Servers are standing next to each other - connected to the same switch Management server runs on the same physical server with host1 I noticed, that Virtual router gets created after I try to start any of the existing guest VM Here are logs management: https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A agent on host1: https://drive.google.com/open?id=1u8YHYIuyU2MA2UKY7G5z7q8p5XxU1zsy agent on host2: https://drive.google.com/open?id=1YzkCL-FmTgPva-QHHp5vTM5Nb3qAXxz4 But this virtual router stays in Starting state forever and hence VMs do not start either. On Wed, Mar 20, 2019 at 2:49 PM Andrija Panic <[email protected]> wrote: > Just to confirm, you are using Basic Zone in CloudStack, right ? > > Can you confirm that router has been actually started/created on KVM side, > again, as requested please post logs (mgmt and agent - and note the time > around which you tried to start VR last time it partially succeeded) - we > can't guess what went wrong without logs. > > I would push more effort solving this one, instead of reinstalling - you > might hit the issue again and then it's no good. > > Can you confirm your zone/pod/cluster/hosts are all in Enabled state, i.e. > not disabled and hosts connected AND both SSVM and CPVM are > connectedUP/green > Is your dashboard in GUI all green - no issues there ? > Can you connect your both KVM hosts can access/mount both Primary and > Secondary Storage > > On Wed, 20 Mar 2019 at 13:15, Jevgeni Zolotarjov <[email protected]> > wrote: > > > After dozen of attempts, the Virtual Router could finally be recreated. > But > > its in eternal Starting status, and console prompts it required upgrade > and > > Version is UNKNOWN > > > > It does not resolve the problem, I cannot move further form this point. > > Any hints? > > > > Or I am condemned to do reinstall cloudstack from scratch? > > > > On Wed, Mar 20, 2019 at 11:08 AM Jevgeni Zolotarjov < > > [email protected]> > > wrote: > > > > > Under this defaultGuestNetwork, I go to Virtual Appliances. There is no > > > VMS - "no data to show" > > > > > > I dont have any network, other than this single default one. > > > > > > I've tried adding new network - Add guest network. But I am not able to > > do > > > so, cause in the wizard popup, it offers empty dropdown with Zones > > > selection. And this wizard doesnt not allow to go further without > > selecting > > > Zone > > > > > > On Wed, Mar 20, 2019 at 10:28 AM Andrija Panic < > [email protected]> > > > wrote: > > > > > >> you need to delete/remove all VMs inside this network (tick the > > "Expunge" > > >> button during VM deletion - if you want to really delete the VMs) in > > order > > >> to be able to delete the network - OR simply attach this VM to another > > >> network, make this new network a DEFAULT one (NIC that is...), and > then > > >> detach from old network - and then effectively your VM was "removed" > > from > > >> old network - after this you should be able to delete the old > network. I > > >> assume some DB incosistencies perhaps, being the reason you can not > > >> restart > > >> the network. > > >> > > >> Did you try restarting some other Network - or deploying a new > network, > > >> spin a VM in it, then again try to restart this new network - does it > > >> work ? > > >> > > >> Andrija > > >> > > >> On Wed, 20 Mar 2019 at 08:58, Jevgeni Zolotarjov < > > [email protected]> > > >> wrote: > > >> > > >> > >>>Stop mgmt, > > >> > >>>Stop all agents > > >> > >>>Restart libvirtd (and check libvirt logs afterwards) > > >> > >>>Start agents > > >> > >>>Start mgmt. > > >> > > > >> > I did that numerous time. Nothing really suspicious > > >> > I can see that systems VMs are running - both in cloudstack console > > and > > >> > with virsh list -all > > >> > > > >> > It is apparently problem with network. > > >> > Is there a way to force recreation of defaultGuestNetwork? or force > > >> > recreation of Virtual Router. > > >> > I am unable to delete network, which is supposed to rebuild network > > with > > >> > its router. Thats the issue > > >> > > > >> > The issue with libvirtd was, that eventually at some point it was > > >> updated > > >> > during 4 months of running, and not rebooted. It still worked. We > had > > to > > >> > add listen_tcp = 1 for libvirtd to start working again. > > >> > > > >> > On Wed, Mar 20, 2019 at 9:49 AM Andrija Panic < > > [email protected]> > > >> > wrote: > > >> > > > >> > > As Sergey suggested... but i would also verify no libvirt issues > or > > >> > storage > > >> > > pool issues - so perhaps: > > >> > > > > >> > > Stop mgmt, > > >> > > Stop all agents > > >> > > Restart libvirtd (and check libvirt logs afterwards) > > >> > > Start agents > > >> > > Start mgmt. > > >> > > > > >> > > What was originally issue with libvirtd ? > > >> > > That sounds fishy to me... > > >> > > > > >> > > Andrija > > >> > > > > >> > > On Wed, Mar 20, 2019, 02:15 Sergey Levitskiy <[email protected] > > > > >> > wrote: > > >> > > > > >> > > > select * from networks where removed is null; > > >> > > > select * from vm_instance where id=87; > > >> > > > select id,name from vm_instance where name like 'r%' and removed > > is > > >> > null; > > >> > > > > > >> > > > Basically since the network offering is not redundant this error > > is > > >> > only > > >> > > > thrown when there is no router associated with your network. > > Usually > > >> > > > management server restart tries to implement network again. > Please > > >> > > restart > > >> > > > management server, save and share management server log. > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > On 3/19/19, 3:31 PM, "Jevgeni Zolotarjov" < > > [email protected]> > > >> > > wrote: > > >> > > > > > >> > > > >>>>Check network_offering table for value in column > > >> > > > redundant_router_service for the network offering you use. > > >> > > > in table network_offering_table all records have > > >> > > > redundant_router_service = > > >> > > > 0 > > >> > > > > > >> > > > Can you also run the following: > > >> > > > >>>select name, state, removed from host where name like > 'r%' > > >> > > > returns zero rows - nothing > > >> > > > > > >> > > > >>>select * from domain_router; > > >> > > > # id, element_id, public_mac_address, public_ip_address, > > >> > > > public_netmask, > > >> > > > guest_netmask, guest_ip_address, is_redundant_router, > > priority, > > >> > > > redundant_state, stop_pending, role, template_version, > > >> > > scripts_version, > > >> > > > vpc_id, update_state > > >> > > > '4', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, > 'UNKNOWN', > > >> '0', > > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 > > 15:37:28 > > >> > UTC > > >> > > > 2018', > > >> > > > '57db7bd8118977a5f2cd3ef1c7503633\n', NULL, NULL > > >> > > > '49', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, > 'UNKNOWN', > > >> '0', > > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 > > 15:37:28 > > >> > UTC > > >> > > > 2018', > > >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > > >> > > > '73', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, > 'UNKNOWN', > > >> '0', > > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 > > 15:37:28 > > >> > UTC > > >> > > > 2018', > > >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > > >> > > > '74', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, > 'UNKNOWN', > > >> '0', > > >> > > > 'VIRTUAL_ROUTER', NULL, NULL, NULL, NULL > > >> > > > '75', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, > 'UNKNOWN', > > >> '0', > > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 > > 15:37:28 > > >> > UTC > > >> > > > 2018', > > >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > > >> > > > '76', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, > 'UNKNOWN', > > >> '0', > > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.0 Sun Jan 14 > > 15:37:28 > > >> > UTC > > >> > > > 2018', > > >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > > >> > > > '77', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, > 'UNKNOWN', > > >> '0', > > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 > > 07:52:17 > > >> > UTC > > >> > > > 2018', > > >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, 'UPDATE_FAILED' > > >> > > > '80', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, > 'UNKNOWN', > > >> '0', > > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 > > 07:52:17 > > >> > UTC > > >> > > > 2018', > > >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > > >> > > > '85', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, > 'UNKNOWN', > > >> '0', > > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.1 Fri Jun 22 > > 07:52:17 > > >> > UTC > > >> > > > 2018', > > >> > > > 'c03a474302d89fa82d345e10fe4cb751\n', NULL, NULL > > >> > > > '86', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, > 'UNKNOWN', > > >> '0', > > >> > > > 'VIRTUAL_ROUTER', NULL, NULL, NULL, NULL > > >> > > > '87', '1', NULL, NULL, NULL, NULL, NULL, '0', NULL, > 'UNKNOWN', > > >> '0', > > >> > > > 'VIRTUAL_ROUTER', 'Cloudstack Release 4.11.2.0 Mon Nov 12 > > >> 15:06:49 > > >> > > UTC > > >> > > > 2018', '873057731ff2cba4a1f3b2411765407c\n', NULL, NULL > > >> > > > > > >> > > > > > >> > > > >>>select * from router_network_ref; > > >> > > > # id, router_id, network_id, guest_type > > >> > > > '1', '4', '204', 'Shared' > > >> > > > '2', '49', '204', 'Shared' > > >> > > > '3', '73', '204', 'Shared' > > >> > > > '4', '75', '204', 'Shared' > > >> > > > '5', '76', '204', 'Shared' > > >> > > > '6', '77', '204', 'Shared' > > >> > > > '7', '80', '204', 'Shared' > > >> > > > '8', '85', '204', 'Shared' > > >> > > > '9', '86', '204', 'Shared' > > >> > > > '10', '87', '204', 'Shared' > > >> > > > > > >> > > > > > >> > > > On Wed, Mar 20, 2019 at 12:18 AM Sergey Levitskiy < > > >> > > [email protected] > > >> > > > > > > >> > > > wrote: > > >> > > > > > >> > > > > Check network_offering table for value in column > > >> > > > > redundant_router_service for the network offering you > use. > > >> > > > > Can you also run the following: > > >> > > > > select name, state, removed from host where name like > 'r%' > > >> > > > > select * from domain_router; > > >> > > > > select * from router_network_ref; > > >> > > > > > > >> > > > > Cloudstack is supposed to recreate you VR. If it is not > > >> happening > > >> > > > there is > > >> > > > > something fundamentally wrong. I would advise to destroy > > your > > >> VR > > >> > > > again. > > >> > > > > Stop you management server. Rotate management server log > and > > >> > start > > >> > > it > > >> > > > > again. If your VR doesn't start in few min, post your > > complete > > >> > > > management > > >> > > > > server log and agent log again. > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > On 3/19/19, 2:56 PM, "Jevgeni Zolotarjov" < > > >> > [email protected]> > > >> > > > wrote: > > >> > > > > > > >> > > > > >>>Network offering needs to be change to > non-redundant > > >> > > > > How do I do that? > > >> > > > > > > >> > > > > On Tue, Mar 19, 2019 at 11:47 PM Sergey Levitskiy < > > >> > > > [email protected] > > >> > > > > > > > >> > > > > wrote: > > >> > > > > > > >> > > > > > Network offering needs to be change to > non-redundant. > > >> Most > > >> > > > likely > > >> > > > > old bug > > >> > > > > > is resurfaced. > > >> > > > > > > https://issues.apache.org/jira/browse/CLOUDSTACK-9024 > > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > On 3/19/19, 2:19 PM, "Jevgeni Zolotarjov" < > > >> > > > [email protected]> > > >> > > > > wrote: > > >> > > > > > > > >> > > > > > Hello > > >> > > > > > > > >> > > > > > I did exactly like you suggested. > > >> > > > > > > > >> > > > > > After UPDATE on db, I can see the router in > > >> cloudstack > > >> > > > console. > > >> > > > > > But attempt to restart network fails. > > >> > > > > > I get error: > > >> > > > > > Resource [DataCenter:1] is unreachable: Can't > find > > >> all > > >> > > > necessary > > >> > > > > > running > > >> > > > > > routers! > > >> > > > > > > > >> > > > > > I rechecked agents on both servers. They look > > >> running > > >> > OK. > > >> > > > > > > > >> > > > > > On Tue, Mar 19, 2019 at 10:00 PM Andrija Panic < > > >> > > > > > [email protected]> > > >> > > > > > wrote: > > >> > > > > > > > >> > > > > > > Ok, so: > > >> > > > > > > > > >> > > > > > > 1.BACKUP YOUR DB - in case there is issues, > you > > >> will > > >> > > > want to > > >> > > > > restore > > >> > > > > > > 2. I tried to reproduce your case on 4.11.2 > > >> (though > > >> > > > didn't do > > >> > > > > any > > >> > > > > > > maintenance etc.) - and could not - I have > > >> > artificially > > >> > > > marked > > >> > > > > > existing VR > > >> > > > > > > as removed in DB - and tried to restart > network > > >> and > > >> > it > > >> > > > worked > > >> > > > > just > > >> > > > > > fine. > > >> > > > > > > > > >> > > > > > > Let's try to "undelete" the VR for that > network > > - > > >> > > though > > >> > > > I > > >> > > > > can't be > > >> > > > > > sure if > > >> > > > > > > this will work or not. > > >> > > > > > > > > >> > > > > > > Find VR based on the network UUID: > > >> > > > > > > SELECT * FROM vm_instance WHERE id IN > > >> (SELECT > > >> > > > > instance_id from > > >> > > > > > nics > > >> > > > > > > where network_id IN (SELECT id FROM networks > > WHERE > > >> > > > > > > UUID="65ca9a05-ff96-4563-ab9c-ffb610dc8b73")); > > >> > > > > > > Obviously, replace this UUID with your network > > >> UUID. > > >> > > > > > > SQL above will return all VMs (user VMs, VRs, > > etc, > > >> > > > delete or > > >> > > > > not...) > > >> > > > > > that > > >> > > > > > > have NIC inside the network with that UUID - > > look > > >> for > > >> > > the > > >> > > > > r-XX-VM > > >> > > > > > being a > > >> > > > > > > router. > > >> > > > > > > > > >> > > > > > > Now that you have the name of the VR (i.e. > > >> r-4-VM) - > > >> > > you > > >> > > > want > > >> > > > > to set > > >> > > > > > 2 > > >> > > > > > > fields to some other value than what it might > be > > >> > > > already, as > > >> > > > > below: > > >> > > > > > > > > >> > > > > > > Need to set field "state" to Stopped: > > >> > > > > > > UPDATE cloud.vm_instance SET > > state='Stopped' > > >> > > WHERE > > >> > > > > > name="r-4-VM"; > > >> > > > > > > (what state is set currently ???) > > >> > > > > > > > > >> > > > > > > Need to set "removed" to NULL value: > > >> > > > > > > UPDATE cloud.vm_instance SET removed = > > NULL > > >> > WHERE > > >> > > > > > NAME="r-4-VM"; (is > > >> > > > > > > there a removed date already set ???) > > >> > > > > > > > > >> > > > > > > Obviously make sure that this VR is not > running > > on > > >> > any > > >> > > > > hypervisor > > >> > > > > > and if > > >> > > > > > > running, virsh destroy it... > > >> > > > > > > > > >> > > > > > > AFTER above has been done (VR is considered to > > >> exist > > >> > > but > > >> > > > > stopped, > > >> > > > > > from > > >> > > > > > > CloudStack point of view) - try to RESTART the > > >> > network > > >> > > > (don't > > >> > > > > bother > > >> > > > > > > deleteing it, since there are VMs in that > > network) > > >> > > > > > > > > >> > > > > > > I'm not 100% positive this will fix your > issue, > > >> but > > >> > > > doesn't > > >> > > > > hurt to > > >> > > > > > try > > >> > > > > > > > > >> > > > > > > If above doesn't work - I would still take a > > look > > >> > into > > >> > > > agents > > >> > > > > and if > > >> > > > > > they > > >> > > > > > > are still connected - optionally, restart > agents > > >> on > > >> > > both > > >> > > > hosts > > >> > > > > once > > >> > > > > > more > > >> > > > > > > and confirm they are connected and up. > > >> > > > > > > > > >> > > > > > > Let us know how it goes (and please backup DB > > once > > >> > more > > >> > > > before > > >> > > > > any > > >> > > > > > actions > > >> > > > > > > !) > > >> > > > > > > > > >> > > > > > > Andrija > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > On Tue, 19 Mar 2019 at 20:41, Jevgeni > > Zolotarjov < > > >> > > > > > [email protected]> > > >> > > > > > > wrote: > > >> > > > > > > > > >> > > > > > > > Yes. Just a single network. > > >> > > > > > > > > > >> > > > > > > > On Tue, 19 Mar 2019, 21:39 Andrija Panic, < > > >> > > > > [email protected] > > >> > > > > > > > > >> > > > > > > wrote: > > >> > > > > > > > > > >> > > > > > > > > Just one more clarification - this is > > Isolate > > >> > > single > > >> > > > > network (not > > >> > > > > > > Shared > > >> > > > > > > > > Network, not VPC) ? > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > On Tue, Mar 19, 2019, 19:36 Jevgeni > > >> Zolotarjov < > > >> > > > > > [email protected] > > >> > > > > > > > > > >> > > > > > > > > wrote: > > >> > > > > > > > > > > >> > > > > > > > > > Name defaultGuestNetwork > > >> > > > > > > > > > ID 4ba834ed-48f3-468f-b667-9bb2d2c258f1 > > >> > > > > > > > > > Zone PBT zone 1 > > >> > > > > > > > > > Description defaultGuestNetwork > > >> > > > > > > > > > Type Shared > > >> > > > > > > > > > State Setup > > >> > > > > > > > > > VPC ID N/A > > >> > > > > > > > > > Persistent No > > >> > > > > > > > > > > > >> > > > > > > > > > On Tue, Mar 19, 2019 at 8:29 PM Andrija > > >> Panic < > > >> > > > > > > [email protected] > > >> > > > > > > > > > > >> > > > > > > > > > wrote: > > >> > > > > > > > > > > > >> > > > > > > > > > > Share the network id and name as seen > > from > > >> > > GUI... > > >> > > > > > > > > > > > > >> > > > > > > > > > > On Tue, Mar 19, 2019, 19:27 Jevgeni > > >> > Zolotarjov > > >> > > < > > >> > > > > > > > [email protected] > > >> > > > > > > > > > > > >> > > > > > > > > > > wrote: > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >>> 1. Confirm please rolling > restart > > is > > >> > set > > >> > > > to false > > >> > > > > > please - > > >> > > > > > > > double > > >> > > > > > > > > > > check > > >> > > > > > > > > > > > Double checked - It is set to false > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > >>>>2. If so - do you know the name > of > > >> VR > > >> > > > which you > > >> > > > > > deleted ? Is > > >> > > > > > > it > > >> > > > > > > > > > last > > >> > > > > > > > > > > > one > > >> > > > > > > > > > > > >>> ever created - if so we can find > > it > > >> > > > easily... > > >> > > > > > > > > > > > I dont know the name > > >> > > > > > > > > > > > Is there a way to fetch it from DB? > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > On Tue, Mar 19, 2019 at 7:58 PM > > Andrija > > >> > > Panic < > > >> > > > > > > > > [email protected] > > >> > > > > > > > > > > > > >> > > > > > > > > > > > wrote: > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > Right...still complaining on > missing > > >> > > running > > >> > > > > routers. > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > 1. Confirm please rolling restart > is > > >> set > > >> > to > > >> > > > false > > >> > > > > please > > >> > > > > > - > > >> > > > > > > double > > >> > > > > > > > > > check > > >> > > > > > > > > > > > > 2. If so - do you know the name of > > VR > > >> > which > > >> > > > you > > >> > > > > deleted > > >> > > > > > ? Is it > > >> > > > > > > > > last > > >> > > > > > > > > > > one > > >> > > > > > > > > > > > > ever created - if so we can find > it > > >> > > easily... > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > On Tue, Mar 19, 2019, 18:40 > Jevgeni > > >> > > > Zolotarjov < > > >> > > > > > > > > > [email protected] > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > wrote: > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Here is management server log > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > >> > > > > > https://drive.google.com/open?id=1H2jI0roeiWxtzReB8qV6QxDkNpaki99A > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > On Tue, Mar 19, 2019 at 7:29 PM > > >> Andrija > > >> > > > Panic < > > >> > > > > > > > > > > [email protected] > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > wrote: > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > I would also try to "undelete" > > VR > > >> in > > >> > > DB, > > >> > > > but > > >> > > > > let's > > >> > > > > > keep > > >> > > > > > > this > > >> > > > > > > > as > > >> > > > > > > > > > > last > > >> > > > > > > > > > > > > > step. > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > On Tue, Mar 19, 2019, 18:24 > > >> Andrija > > >> > > > Panic < > > >> > > > > > > > > > [email protected] > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > wrote: > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > Disclaimer: from mobile > > device. > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > I don't see any special > > failure > > >> in > > >> > > > agent log, > > >> > > > > > except some > > >> > > > > > > > > long > > >> > > > > > > > > > > > > running > > >> > > > > > > > > > > > > > > > migration, timeout for > > graceful > > >> VM > > >> > > > shutdown > > >> > > > > etc > > >> > > > > > (and > > >> > > > > > > agent > > >> > > > > > > > > > > restart) > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > You did not send mgmt log > > after > > >> > last > > >> > > > network > > >> > > > > > restart > > >> > > > > > > > failure > > >> > > > > > > > > ? > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > On Tue, Mar 19, 2019, 17:29 > > >> Jevgeni > > >> > > > > Zolotarjov < > > >> > > > > > > > > > > > > [email protected] > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > wrote: > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> Under > > >> > > > > > > > > > > > > > > >> Infrastructure / Hosts: > > >> > > > > > > > > > > > > > > >> both hosts are enabled and > > >> green > > >> > > > (Unsecure > > >> > > > > though) > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> agent.log - > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > >> > > > > > > > > >> > > > > > > >> > https://drive.google.com/open?id=1rATxHKqgNKo2kD23BtlrZy_9gFXC-Bq- > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> I set > network.rolling.restart > > >> to > > >> > > > false now. > > >> > > > > > Restarted > > >> > > > > > > > > > management > > >> > > > > > > > > > > > > > server > > >> > > > > > > > > > > > > > > - > > >> > > > > > > > > > > > > > > >> same problem - cannot > restart > > >> not > > >> > > > delete > > >> > > > > network > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> On Tue, Mar 19, 2019 at > 5:48 > > PM > > >> > > Boris > > >> > > > > Stoyanov < > > >> > > > > > > > > > > > > > > >> > [email protected] > > > > > >> > > > > > > > > > > > > > > >> wrote: > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > >> > Hi, you shouldn’t be > > worried > > >> > about > > >> > > > data > > >> > > > > as long > > >> > > > > > as you > > >> > > > > > > > > use a > > >> > > > > > > > > > > > > > separate > > >> > > > > > > > > > > > > > > >> > shared storage. > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > What does the cloudstack > > >> console > > >> > > say > > >> > > > > about your > > >> > > > > > host? > > >> > > > > > > Is > > >> > > > > > > > > it > > >> > > > > > > > > > > up? > > >> > > > > > > > > > > > If > > >> > > > > > > > > > > > > > > it’s > > >> > > > > > > > > > > > > > > >> up > > >> > > > > > > > > > > > > > > >> > then you should be able > to > > >> > deploy > > >> > > a > > >> > > > VM on > > >> > > > > it. > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > If not, you’ll need to > > >> > investigate > > >> > > > the > > >> > > > > reason > > >> > > > > > > > > > cloudstack-agent > > >> > > > > > > > > > > > is > > >> > > > > > > > > > > > > > not > > >> > > > > > > > > > > > > > > >> > connected, is the agent > > >> running? > > >> > > If > > >> > > > not > > >> > > > > just > > >> > > > > > start it > > >> > > > > > > > and > > >> > > > > > > > > > see > > >> > > > > > > > > > > if > > >> > > > > > > > > > > > > it > > >> > > > > > > > > > > > > > > >> > connects. > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > If it’s running but still > > not > > >> > > > connected > > >> > > > > please > > >> > > > > > share > > >> > > > > > > the > > >> > > > > > > > > > > > > agent.log. > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > If nothing helps at all > you > > >> can > > >> > > > remove > > >> > > > > the host > > >> > > > > > and > > >> > > > > > > add > > >> > > > > > > > it > > >> > > > > > > > > > > > again, > > >> > > > > > > > > > > > > > but > > >> > > > > > > > > > > > > > > >> > let’s try to troubleshoot > > >> first. > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > Bobby. > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > > > [email protected] > > >> > > > > > > > > > > > > > > >> > www.shapeblue.com > > >> > > > > > > > > > > > > > > >> > Amadeus House, Floral > > Street, > > >> > > London > > >> > > > > WC2E 9DPUK > > >> > > > > > > > > > > > > > > >> > @shapeblue > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > > On 19 Mar 2019, at > 17:29, > > >> > > Jevgeni > > >> > > > > Zolotarjov < > > >> > > > > > > > > > > > > > > [email protected]> > > >> > > > > > > > > > > > > > > >> > wrote: > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > >> > > Guys, please help with > > it. > > >> > > > > > > > > > > > > > > >> > > What can be done here? > > >> > > > > > > > > > > > > > > >> > > There is too much > > valuable > > >> > data. > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > >> > > On Tue, Mar 19, 2019 at > > >> 4:21 > > >> > PM > > >> > > > Jevgeni > > >> > > > > > Zolotarjov < > > >> > > > > > > > > > > > > > > >> > [email protected]> > > >> > > > > > > > > > > > > > > >> > > wrote: > > >> > > > > > > > > > > > > > > >> > > > > >> > > > > > > > > > > > > > > >> > >> Tried that just now > and > > >> got > > >> > > > error: > > >> > > > > > > > > > > > > > > >> > >> Resource > [DataCenter:1] > > is > > >> > > > unreachable: > > >> > > > > > Can't find > > >> > > > > > > > all > > >> > > > > > > > > > > > > necessary > > >> > > > > > > > > > > > > > > >> running > > >> > > > > > > > > > > > > > > >> > >> routers! > > >> > > > > > > > > > > > > > > >> > >> > > >> > > > > > > > > > > > > > > >> > >> In the log I see: > > >> > > > > > > > > > > > > > > >> > >> > > >> > > > > ============================================= > > >> > > > > > > > > > > > > > > >> > >> > > >> > > > > > > > > > > > > > > >> > >> 2019-03-19 > 14:20:39,644 > > >> DEBUG > > >> > > > > > > > > > > [o.a.c.e.o.NetworkOrchestrator] > > >> > > > > > > > > > > > > > > >> > >> > > >> > > (API-Job-Executor-4:ctx-7b6b69eb > > >> > > > > job-5093 > > >> > > > > > > > ctx-9be30648) > > >> > > > > > > > > > > > > > > >> (logid:265a6099) > > >> > > > > > > > > > > > > > > >> > >> Restarting network > > 204... > > >> > > > > > > > > > > > > > > >> > >> 2019-03-19 > 14:20:39,645 > > >> DEBUG > > >> > > > > > > > > > > [o.a.c.e.o.NetworkOrchestrator] > > >> > > > > > > > > > > > > > > >> > >> > > >> > > (API-Job-Executor-4:ctx-7b6b69eb > > >> > > > > job-5093 > > >> > > > > > > > ctx-9be30648) > > >> > > > > > > > > > > > > > > >> (logid:265a6099) > > >> > > > > > > > > > > > > > > >> > >> Performing rolling > > >> restart of > > >> > > > routers > > >> > > > > of > > >> > > > > > network > > >> > > > > > > > > > > > > > Ntwk[204|Guest|6] > > >> > > > > > > > > > > > > > > >> > >> 2019-03-19 > 14:20:39,649 > > >> DEBUG > > >> > > > > > > > > > > [o.a.c.e.o.NetworkOrchestrator] > > >> > > > > > > > > > > > > > > >> > >> > > >> > > (API-Job-Executor-4:ctx-7b6b69eb > > >> > > > > job-5093 > > >> > > > > > > > ctx-9be30648) > > >> > > > > > > > > > > > > > > >> (logid:265a6099) > > >> > > > > > > > > > > > > > > >> > >> Asking VirtualRouter > to > > >> > > > implemenet > > >> > > > > > > Ntwk[204|Guest|6] > > >> > > > > > > > > > > > > > > >> > >> 2019-03-19 > 14:20:39,658 > > >> ERROR > > >> > > > > > > > > > [c.c.a.ApiAsyncJobDispatcher] > > >> > > > > > > > > > > > > > > >> > >> > > >> > > (API-Job-Executor-4:ctx-7b6b69eb > > >> > > > > job-5093) > > >> > > > > > > > > > (logid:265a6099) > > >> > > > > > > > > > > > > > > >> Unexpected > > >> > > > > > > > > > > > > > > >> > >> exception while > > executing > > >> > > > > > > > > > > > > > > >> > >> > > >> > > > > > > > > > > > > > > >> > > > > > > > > >> > > > > > > >> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd > > >> > > > > > > > > > > > > > > >> > >> > > >> > > > > > com.cloud.exception.ResourceUnavailableException: > > >> > > > > > > > > > Resource > > >> > > > > > > > > > > > > > > >> > [DataCenter:1] > > >> > > > > > > > > > > > > > > >> > >> is unreachable: Can't > > find > > >> > all > > >> > > > > necessary > > >> > > > > > running > > >> > > > > > > > > routers! > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > > > > > > > > > >> > >> > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:243) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > > > > > > > > > >> > >> > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1203) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > > > > > > > > > >> > >> > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.rollingRestartRouters(NetworkOrchestrator.java:2948) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > > > > > > > > > >> > >> > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2853) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > > > > > > > > > >> > >> > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > > > > > > > >> > > > sun.reflect.NativeMethodAccessorImpl.invoke0(Native > > >> > > > > > > > > > > > > > > Method) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > > > > > > > >> > > > sun.reflect.NativeMethodAccessorImpl.invoke(Unknown > > >> > > > > > > > > > > > > > > Source) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > > > > > > > > > >> > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > > >> > > > > > > > > > > > > > > >> > Source) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > java.lang.reflect.Method.invoke(Unknown > > >> > > > > > > > > Source) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > > > > > > > > > >> > >> > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > > > > > > > > > >> > >> > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > > > > > > > > > >> > >> > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > > > > > > > > > >> > >> > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > > > > > > > > > >> > >> > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > > > > > > > > > >> > >> > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > > > > > > > > > >> > >> > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > > > > > > > > > >> > >> > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > > > > > > > > > >> > >> > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > > > > > > > > > >> > >> > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > > > > > com.sun.proxy.$Proxy229.restartNetwork(Unknown > > >> > > > > > > > > > > > > Source) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > > > > > > > > > >> > >> > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > >> > > > com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > > > > > > > > > >> > >> > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > > > > > > > > > >> > >> > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > > > > > > > > > >> > >> > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > > > > > > > > > >> > >> > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > > > > > > > > > >> > >> > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > > > > > > > > > >> > >> > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > > > > > > > > > >> > >> > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > > > > > > > > > >> > >> > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > > > > > > > > > > > >> > > > > > > > >> java.util.concurrent.Executors$RunnableAdapter.call(Unknown > > >> > > > > > > > > > > > > > > >> > >> Source) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > java.util.concurrent.FutureTask.run(Unknown > > >> > > > > > > > > > > Source) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > > > > > > > > > > >> > > > > > > > >> java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown > > >> > > > > > > > > > > > > > > >> > >> Source) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > > > > > > > > > > > > > > >> > > > > > > > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > > >> > > > > > > > > > > > > > > >> > >> Source) > > >> > > > > > > > > > > > > > > >> > >> at > > >> > > > java.lang.Thread.run(Unknown > > >> > > > > > Source) > > >> > > > > > > > > > > > > > > >> > >> 2019-03-19 > 14:20:39,659 > > >> DEBUG > > >> > > > > > > > > > > > [o.a.c.f.j.i.AsyncJobManagerImpl] > > >> > > > > > > > > > > > > > > >> > >> > > >> > > (API-Job-Executor-4:ctx-7b6b69eb > > >> > > > > job-5093) > > >> > > > > > > > > > (logid:265a6099) > > >> > > > > > > > > > > > > > > Complete > > >> > > > > > > > > > > > > > > >> > async > > >> > > > > > > > > > > > > > > >> > >> job-5093, jobStatus: > > >> FAILED, > > >> > > > > resultCode: 530, > > >> > > > > > > result: > > >> > > > > > > > > > > > > > > >> > >> > > >> > > > > > > > > > > > > > > >> > > > >> > > > > > > > > > > > > > > >> > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Resource > > >> > > > > > > > > > > > > > > >> > >> [DataCenter:1] is un > > >> > > > > > > > > > > > > > > >> > >> > > >> > > > > > > > > > > > > > > >> > >> On Tue, Mar 19, 2019 > at > > >> 4:19 > > >> > PM > > >> > > > Andrija > > >> > > > > > Panic < > > >> > > > > > > > > > > > > > > >> > > > [email protected]> > > >> > > > > > > > > > > > > > > >> > >> wrote: > > >> > > > > > > > > > > > > > > >> > >> > > >> > > > > > > > > > > > > > > >> > >>> > > >> > > > > > > > > > > > > > > >> > >>> Your network can't be > > >> > deleted > > >> > > > due to > > >> > > > > "Can't > > >> > > > > > delete > > >> > > > > > > > the > > >> > > > > > > > > > > > > network, > > >> > > > > > > > > > > > > > > not > > >> > > > > > > > > > > > > > > >> all > > >> > > > > > > > > > > > > > > >> > >>> user vms are > expunged. > > Vm > > >> > > > > > > > > > > > > > > >> > >>> VM[User|i-2-11-VM] is > > in > > >> > > Stopped > > >> > > > > state" - > > >> > > > > > which is > > >> > > > > > > > > fine. > > >> > > > > > > > > > > > > > > >> > >>> > > >> > > > > > > > > > > > > > > >> > >>> You should be able to > > >> just > > >> > > > start the > > >> > > > > user > > >> > > > > > VM - but > > >> > > > > > > > if > > >> > > > > > > > > > you > > >> > > > > > > > > > > > have > > >> > > > > > > > > > > > > > > >> actually > > >> > > > > > > > > > > > > > > >> > >>> delete the VR itself, > > >> then > > >> > > just > > >> > > > do > > >> > > > > Network > > >> > > > > > restart > > >> > > > > > > > > with > > >> > > > > > > > > > > > > > "cleanup" > > >> > > > > > > > > > > > > > > >> and > > >> > > > > > > > > > > > > > > >> > it > > >> > > > > > > > > > > > > > > >> > >>> will recreate a new > VR, > > >> > after > > >> > > > which > > >> > > > > you > > >> > > > > > should be > > >> > > > > > > > able > > >> > > > > > > > > > to > > >> > > > > > > > > > > > > start > > >> > > > > > > > > > > > > > > the > > >> > > > > > > > > > > > > > > >> VM. > > >> > > > > > > > > > > > > > > >> > >>> > > >> > > > > > > > > > > > > > > >> > >>> Andrija > > >> > > > > > > > > > > > > > > >> > >>> > > >> > > > > > > > > > > > > > > >> > >>> > > >> [email protected] > > >> > > > > > > > > > > > > > > >> > >>> www.shapeblue.com > > >> > > > > > > > > > > > > > > >> > >>> Amadeus House, Floral > > >> > Street, > > >> > > > London > > >> > > > > WC2E > > >> > > > > > 9DPUK > > >> > > > > > > > > > > > > > > >> > >>> @shapeblue > > >> > > > > > > > > > > > > > > >> > >>> > > >> > > > > > > > > > > > > > > >> > >>> > > >> > > > > > > > > > > > > > > >> > >>> > > >> > > > > > > > > > > > > > > >> > >>> > > >> > > > > > > > > > > > > > > >> > >>> -----Original > > >> Message----- > > >> > > > > > > > > > > > > > > >> > >>> From: Jevgeni > > Zolotarjov > > >> < > > >> > > > > > [email protected]> > > >> > > > > > > > > > > > > > > >> > >>> Sent: 19 March 2019 > > 15:10 > > >> > > > > > > > > > > > > > > >> > >>> To: > > >> > > [email protected] > > >> > > > > > > > > > > > > > > >> > >>> Subject: Re: Disaster > > >> after > > >> > > > > maintenance > > >> > > > > > > > > > > > > > > >> > >>> > > >> > > > > > > > > > > > > > > >> > >>> I mean I cannot > delete > > >> > > network: > > >> > > > In the > > >> > > > > > management > > >> > > > > > > > > server > > >> > > > > > > > > > > > log I > > >> > > > > > > > > > > > > > see > > >> > > > > > > > > > > > > > > >> > >>> > > >> > > > > > > > > > > > > > > >> > >>> > > >> > > > > > > > > > > >> > > > > >> > > > >> > > >> > > >> -- > > >> > > >> Andrija Panić > > >> > > > > > > > > -- > > Andrija Panić >
