Hi Chris, You can try to update the network/vpc tier to an offering with Userdata and then disable static nat. May I ask why you decided not to use Userdata ?
-Wei On Tue, 4 Jan 2022 at 14:35, [email protected] <[email protected]> wrote: > As i will need to fix this - i guess somewhat during the reset of the week > are there any ideas on how to "remove" this from CS? (BackUP Jobs will be > started for DB and i wouldn't have the chance to perform a binarylog > rollback / recovery) > > The "clean" way i guess would be a DB Rollback to a point prior to > enableing static NAT. > Taking a look into the DB itself, i only found one entry in the table > "user_ip_address": Entry with rulestate / state "Releasing" for the ip is > present. one_to_one_nat = 1 and the dnat_vmip is present. > Does anyone has some experience on cleaning this up directly in db? > > Regards, > Chris > > Am Di., 4. Jan. 2022 um 13:16 Uhr schrieb [email protected] <[email protected]>: > > > Hi Wei, > > > > of course! :-) > > > > https://github.com/apache/cloudstack/issues/5824 > > > > Regards, > > Chris > > > > Am Di., 4. Jan. 2022 um 11:32 Uhr schrieb Wei ZHOU < > [email protected] > > >: > > > >> Hi Chris, > >> > >> This seems to be a bug. Can you file an issue on > >> https://github.com/apache/cloudstack/issues ? > >> > >> -Wei > >> > >> On Tue, 4 Jan 2022 at 11:16, [email protected] <[email protected]> wrote: > >> > >>> Hi Dan, > >>> > >>> Sorry for the misunderstanding. > >>> The VPC was created using a "custome" service offering which is not > >>> using / > >>> offering "service user data" . The same goes for the NW-Offering which > is > >>> used for creation of the tiers. There is no service user data in this > >>> VPC. > >>> > >>> The default / out-of-the-box VPC offering has the service "user data" > >>> included. Same goes for the default nework offering for VPC Tiers. So > >>> this > >>> service should be supported at all from my understanding. > >>> That's why this error seems a bit "bogus" to me. I was understanding > >>> somethin like "static nat can't be disabled due to missing service > "user > >>> data" for the network". > >>> > >>> Deleting and building the whole thing from scratch will be the last > >>> resort, > >>> yes. Even taking the risk that it won't be deletable as the allocated > >>> ressourcces can't be released properly at the moment, e.g. using the > >>> cloudstack.api.command.user.nat.DisableStaticNatCmd. > >>> Seems like i found a configuration which doesn't work well but can be > >>> created via the GUI. Even if i don't understand the connection between > >>> the > >>> service user data and the service stativ nat. > >>> > >>> > >>> > >>> Daan Hoogland <[email protected]> schrieb am Di., 4. Jan. 2022, > >>> 09:21: > >>> > >>> > Chris, it looks to me like you should delete the VPC and then check > if > >>> all > >>> > IPs are available before creating a new one. > >>> > > >>> > As for the user data service, the message already says that user data > >>> is > >>> > not supported, so why create another offering? > >>> > > >>> > On Tue, Jan 4, 2022 at 2:50 AM [email protected] <[email protected]> wrote: > >>> > > >>> >> hi everyone, > >>> >> > >>> >> sadly i am encountering some problems in my CS 4.16 set-up. > >>> >> > >>> >> today i wanted to take a look into static-nat for the routers of my > >>> vpc. > >>> >> Sadly i missed, that the VM i wanted to assaign to the static public > >>> ip > >>> >> had > >>> >> was already targeted by an port forwarding. > >>> >> As you can imagine i got an error that static can't be applied for > >>> this VM > >>> >> / default interface. > >>> >> > >>> >> Thats when strange things happend. > >>> >> The static NAT configuration was successfully applied. > >>> >> I wanted to revert everything so i tried to delete the static NAT > And > >>> i > >>> >> got > >>> >> the following error: > >>> >> > >>> >> 2022-01-04 00:55:58,394 DEBUG [o.a.c.n.t.AdvancedNetworkTopology] > >>> >> (API-Job-Executor-98:ctx-90ac5cad job-2135 ctx-2eb15c74) > >>> (logid:90c06d44) > >>> >> APPLYING VPC IP RULES > >>> >> 2022-01-04 00:55:58,399 DEBUG [o.a.c.n.t.BasicNetworkTopology] > >>> >> (API-Job-Executor-98:ctx-90ac5cad job-2135 ctx-2eb15c74) > >>> (logid:90c06d44) > >>> >> Applying vpc ip association in network Ntwk[246|Guest|30] > >>> >> 2022-01-04 00:55:58,442 DEBUG [c.c.a.t.Request] > >>> >> (API-Job-Executor-98:ctx-90ac5cad job-2135 ctx-2eb15c74) > >>> (logid:90c06d44) > >>> >> Seq 5-4862198747700030042: Sending { Cmd , MgmtId: 90520740759984, > >>> via: > >>> >> 5(srv-4-comp.dnd-digital.infra), Ver: v1, Flags: 100001, > >>> >> > >>> >> > >>> > [{"com.cloud.agent.api.routing.IpAssocVpcCommand":{"ipAddresses":[{"accountId":"5","publicIp":"10.129.0.37","sourceNat":"true","add":"false","oneToOneNat":"true","firstIP":"true","broadcastUri":"vlan://2990","vlanGateway":"10.129.0.1","vlanNetmask":"255.255.255.128","vifMacAddress":"1e:00:59:00:00:b2","networkRate":"-1","trafficType":"Public","networkName":"br3_CS_Pub","newNic":"false","isPrivateGateway":"false"}],"accessDetails":{" > >>> >> router.name > >>> >> > >>> > ":"r-62-VM","router.guest.ip":"10.129.0.42","network.public.last.ip":"false","router.ip":"169.254.1.55","zone.network.type":"Advanced"},"wait":"0","bypassHostMaintenance":"false"}}] > >>> >> } > >>> >> 2022-01-04 00:55:59,476 DEBUG [c.c.a.t.Request] > >>> >> (AgentManager-Handler-3:null) (logid:) Seq 5-4862198747700030042: > >>> >> Processing: { Ans: , MgmtId: 90520740759984, via: 5, Ver: v1, > Flags: > >>> 0, > >>> >> [{"com.cloud.agent.api.routing.GroupAnswer":{"results":["null - > >>> success: > >>> >> Creating file in VR, with ip: 169.254.1.55, file: > >>> >> ip_associations.json.167dc5ba-c89a-48c8-9b99-81b375e59691","null - > >>> >> success: > >>> >> Error: Nexthop has invalid gateway. > >>> >> Error: Nexthop has invalid gateway. > >>> >> Error: Nexthop has invalid gateway. > >>> >> Error: Nexthop has invalid gateway. > >>> >> Error: Nexthop has invalid gateway. > >>> >> Error: Nexthop has invalid gateway. > >>> >> Error: Nexthop has invalid gateway. > >>> >> Error: Nexthop has invalid gateway. > >>> >> Error: Nexthop has invalid gateway. > >>> >> Error: Nexthop has invalid gateway. > >>> >> Error: Nexthop has invalid gateway. > >>> >> Error: Nexthop has invalid gateway. > >>> >> Error: Nexthop has invalid gateway. > >>> >> Error: Nexthop has invalid gateway. > >>> >> Error: Nexthop has invalid gateway. > >>> >> Error: Nexthop has invalid gateway. > >>> >> Error: Nexthop has invalid gateway. > >>> >> Error: Nexthop has invalid gateway. > >>> >> Error: Nexthop has invalid gateway. > >>> >> Error: Nexthop has invalid gateway. > >>> >> Error: Nexthop has invalid gateway. > >>> >> Error: Nexthop has invalid gateway. > >>> >> Error: Nexthop has invalid gateway. > >>> >> "],"result":"true","wait":"0","bypassHostMaintenance":"false"}}] } > >>> >> 2022-01-04 00:55:59,476 DEBUG [c.c.a.t.Request] > >>> >> (API-Job-Executor-98:ctx-90ac5cad job-2135 ctx-2eb15c74) > >>> (logid:90c06d44) > >>> >> Seq 5-4862198747700030042: Received: { Ans: , MgmtId: > 90520740759984, > >>> >> via: > >>> >> 5(srv-4-comp.dnd-digital.infra), Ver: v1, Flags: 0, { GroupAnswer } > } > >>> >> 2022-01-04 00:55:59,486 ERROR [c.c.a.ApiAsyncJobDispatcher] > >>> >> (API-Job-Executor-98:ctx-90ac5cad job-2135) (logid:90c06d44) > >>> Unexpected > >>> >> exception while executing > >>> >> org.apache.cloudstack.api.command.user.nat.DisableStaticNatCmd > >>> >> com.cloud.exception.UnsupportedServiceException: Service UserData is > >>> not > >>> >> supported in the network id=246 > >>> >> at > >>> >> > >>> >> > >>> > com.cloud.network.dao.NetworkServiceMapDaoImpl.getProviderForServiceInNetwork(NetworkServiceMapDaoImpl.java:126) > >>> >> at jdk.internal.reflect.GeneratedMethodAccessor1754.invoke(Unknown > >>> Source) > >>> >> at > >>> >> > >>> >> > >>> > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >>> >> at java.base/java.lang.reflect.Method.invoke(Method.java:566) > >>> >> at > >>> >> > >>> >> > >>> > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) > >>> >> at > >>> >> > >>> >> > >>> > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) > >>> >> at > >>> >> > >>> >> > >>> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > >>> >> at > >>> >> > >>> >> > >>> > com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34) > >>> >> at > >>> >> > >>> >> > >>> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) > >>> >> at > >>> >> > >>> >> > >>> > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) > >>> >> at > >>> >> > >>> >> > >>> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) > >>> >> at > >>> >> > >>> >> > >>> > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) > >>> >> at com.sun.proxy.$Proxy61.getProviderForServiceInNetwork(Unknown > >>> Source) > >>> >> at > >>> >> > >>> >> > >>> > com.cloud.network.NetworkModelImpl.getUserDataUpdateProvider(NetworkModelImpl.java:902) > >>> >> at > >>> >> > >>> >> > >>> > com.cloud.network.rules.RulesManagerImpl.applyUserData(RulesManagerImpl.java:633) > >>> >> at > >>> >> > >>> >> > >>> > com.cloud.network.rules.RulesManagerImpl.revokeAllPFAndStaticNatRulesForIp(RulesManagerImpl.java:1147) > >>> >> at > >>> >> > >>> >> > >>> > com.cloud.network.rules.RulesManagerImpl.disableStaticNat(RulesManagerImpl.java:1328) > >>> >> at > >>> >> > >>> >> > >>> > com.cloud.network.rules.RulesManagerImpl.disableStaticNat(RulesManagerImpl.java:1290) > >>> >> at > >>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > >>> >> Method) > >>> >> at > >>> >> > >>> >> > >>> > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >>> >> at > >>> >> > >>> >> > >>> > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >>> >> at java.base/java.lang.reflect.Method.invoke(Method.java:566) > >>> >> at > >>> >> > >>> >> > >>> > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) > >>> >> at > >>> >> > >>> >> > >>> > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) > >>> >> at > >>> >> > >>> >> > >>> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > >>> >> at > >>> >> > >>> >> > >>> > org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) > >>> >> at > >>> >> > >>> >> > >>> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) > >>> >> at > >>> >> > >>> >> > >>> > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) > >>> >> at > >>> >> > >>> >> > >>> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) > >>> >> at > >>> >> > >>> >> > >>> > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) > >>> >> at > >>> >> > >>> >> > >>> > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) > >>> >> at > >>> >> > >>> >> > >>> > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) > >>> >> at com.sun.proxy.$Proxy122.disableStaticNat(Unknown Source) > >>> >> at > >>> >> > >>> >> > >>> > org.apache.cloudstack.api.command.user.nat.DisableStaticNatCmd.execute(DisableStaticNatCmd.java:88) > >>> >> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156) > >>> >> at > >>> >> > >>> > com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) > >>> >> at > >>> >> > >>> >> > >>> > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620) > >>> >> at > >>> >> > >>> >> > >>> > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48) > >>> >> at > >>> >> > >>> >> > >>> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) > >>> >> at > >>> >> > >>> >> > >>> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102) > >>> >> at > >>> >> > >>> >> > >>> > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) > >>> >> at > >>> >> > >>> >> > >>> > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45) > >>> >> at > >>> >> > >>> >> > >>> > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568) > >>> >> at > >>> >> > >>> >> > >>> > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > >>> >> at > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > >>> >> at > >>> >> > >>> >> > >>> > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > >>> >> at > >>> >> > >>> >> > >>> > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > >>> >> at java.base/java.lang.Thread.run(Thread.java:829) > >>> >> 2022-01-04 00:55:59,488 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > >>> >> (API-Job-Executor-98:ctx-90ac5cad job-2135) (logid:90c06d44) > Complete > >>> >> async > >>> >> job-2135, jobStatus: FAILED, resultCode: 530, result: > >>> >> > >>> >> > >>> > org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Service > >>> >> UserData is not supported in the network id=246"} > >>> >> 2022-01-04 00:55:59,489 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > >>> >> (API-Job-Executor-98:ctx-90ac5cad job-2135) (logid:90c06d44) Publish > >>> async > >>> >> job-2135 complete on message bus > >>> >> 2022-01-04 00:55:59,489 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > >>> >> (API-Job-Executor-98:ctx-90ac5cad job-2135) (logid:90c06d44) Wake up > >>> jobs > >>> >> related to job-2135 > >>> >> 2022-01-04 00:55:59,489 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > >>> >> (API-Job-Executor-98:ctx-90ac5cad job-2135) (logid:90c06d44) Update > db > >>> >> status for job-2135 > >>> >> 2022-01-04 00:55:59,490 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > >>> >> (API-Job-Executor-98:ctx-90ac5cad job-2135) (logid:90c06d44) Wake up > >>> jobs > >>> >> joined with job-2135 and disjoin all subjobs created from job- 2135 > >>> >> 2022-01-04 00:55:59,494 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > >>> >> (API-Job-Executor-98:ctx-90ac5cad job-2135) (logid:90c06d44) Done > >>> >> executing > >>> >> org.apache.cloudstack.api.command.user.nat.DisableStaticNatCmd for > >>> >> job-2135 > >>> >> 2022-01-04 00:55:59,494 INFO [o.a.c.f.j.i.AsyncJobMonitor] > >>> >> (API-Job-Executor-98:ctx-90ac5cad job-2135) (logid:90c06d44) Remove > >>> >> job-2135 from job monitoring > >>> >> > >>> >> As far as i understand there is a problem with the "service user > >>> data". > >>> >> That's correct as i created my own offerings for VPC and the > >>> Tier-Networks > >>> >> which don't offer the service. > >>> >> How ever now i have the situation that i can't dele the static nat > >>> via the > >>> >> GUI and the public ip is stuck in the state "released", is not > >>> availeable > >>> >> for further usage and the static NAT is still shown as configured. > >>> >> > >>> >> Maybe someone can point me on how to release the IP properly? May i > >>> need > >>> >> to > >>> >> work directly in the DB to clean this "mess" ? > >>> >> Any suggestion is appreciated. > >>> >> > >>> >> With regards, > >>> >> Chris > >>> >> > >>> > > >>> > > >>> > -- > >>> > Daan > >>> > > >>> > >> >
