kiranchavala commented on issue #10012: URL: https://github.com/apache/cloudstack/issues/10012#issuecomment-2516911833
@DaanHoogland To provide further context on this 1. Once a Kubernetes cluster is deployed on the vmware by cloudstack 2. End user downloads the kubernetes config to interact with the kubernetes cluster via the kubectl tool  3. End user populates the kubeconfig file so that kubectl tool can interact with the cluster vi .kube/config 4. Check the status of the cluster and nodes ``` kubectl get nodes NAME STATUS ROLES AGE VERSION test-control-19377251026 Ready control-plane 4d22h v1.28.4 test-node-19377261815 Ready <none> 4d22h v1.28.4 ``` 5. Deploy a sample application (example: nginx) on kubernetes cluster kubectl apply -f nginx.yaml ``` apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment spec: selector: matchLabels: app: nginx replicas: 2 # tells deployment to run 2 pods matching the template template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.14.2 ports: - containerPort: 80 ``` 6. Check the application status ``` kubectl get pods NAME READY STATUS RESTARTS AGE nginx-deployment-86dcfdf4c6-fvjxp 1/1 Running 0 66m nginx-deployment-86dcfdf4c6-jqjxw 1/1 Running 0 66m ``` 7. Now end user want to access the application via public IP ( public IP range ) which is provided by cloudstack `kubectl expose deploy/nginx-deployment --port=80 --type=LoadBalancer ` 8. Now check the external public IP , it will be struck in pending state ``` k get svc NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 4d22h nginx-deployment LoadBalancer 10.104.200.145 <pending> 80:32333/TCP 66m ``` 9. This is because the controller pods( cloud-controller-manager and kube-controller-manager) which are responsible for assigning a public IP address to the application fails ``` kube-system kube-controller-manager-test-control-19377251026 1/1 Running 79 (6h15m ago) 4d22h 172.30.1.109 test-control-19377251026 kube-system cloud-controller-manager-574bcb86c-9fcgd 1/1 Running 77 (8h ago) 5d1h ``` 10. On checking the Logs of the cloud-controller pod, we can see a timeout issue. k logs -f cloud-controller-manager-574bcb86c-9fcgd -n kube-system ``` 47DXsoV%2BOnK945EvKkfLmLj4tU%3D: dial tcp 10.1.35.76:8080: i/o timeout, and error by node name: error retrieving node addresses: Get http://10.1.35.76:8080/client/api?apiKey=m8GvJ-HIcVY9XurofyAvOLn6mV7aAURljtgpCVH13b5O48ej_ewMNjNo32iKe64oeSuYrzI_gEgg7JCTo_UjPA&command=listVirtualMachines&name=test-control-19377251026&response=json&signature=N6Ykg6Es%2BQWJYXIQ5KKXjyAHoiI%3D: dial tcp 10.1.35.76:8080: i/o timeout E1204 07:44:13.663811 1 controller.go:244] error processing service kube-system/nginx-lb (will retry): failed to ensure load balancer: error retrieving load balancer rules: Get http://10.1.35.76:8080/client/api?apiKey=m8GvJ-HIcVY9XurofyAvOLn6mV7aAURljtgpCVH13b5O48ej_ewMNjNo32iKe64oeSuYrzI_gEgg7JCTo_UjPA&command=listLoadBalancerRules&keyword=ab9e4156d8f724ccfa61f379a8aaa0a6&listall=true&response=json&signature=MYL8q14GpEp5QxnRDg4nRXnkhQg%3D: dial tcp 10.1.35.76:8080: i/o timeout I1204 07:44:13.664131 1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"nginx-lb", UID:"b9e4156d-8f72-4ccf-a61f-379a8aaa0a63", APIVersion:"v1", ResourceVersion:"937", FieldPath:""}): type: 'Warning' reason: 'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load balancer: error retrieving load balancer rules: Get http://10.1.35.76:8080/client/api?apiKey=m8GvJ-HIcVY9XurofyAvOLn6mV7aAURljtgpCVH13b5O48ej_ewMNjNo32iKe64oeSuYrzI_gEgg7JCTo_UjPA&command=listLoadBalancerRules&keyword=ab9e4156d8f724ccfa61f379a8aaa0a6&listall=true&response=json&signature=MYL8q14GpEp5QxnRDg4nRXnkhQg%3D: dial tcp 10.1.35.76:8080: i/o timeout I1204 07:44:13.665040 1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"nginx-deployment", UID:"efa6bff6-3d27-411f-a5be-2a94f7416df7", APIVersion:"v1", ResourceVersion:"995877", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer E1204 07:44:39.309010 1 node_controller.go:237] error retrieving instance ID: Get http://10.1.35.76:8080/client/api?apiKey=m8GvJ-HIcVY9XurofyAvOLn6mV7aAURljtgpCVH13b5O48ej_ewMNjNo32iKe64oeSuYrzI_gEgg7JCTo_UjPA&command=listVirtualMachines&name=test-node-19377261815&response=json&signature=1ONErKvq5OZxjpR26hzPmd8PLtY%3D: dial tcp 10.1.35.76:8080: i/o timeout E1204 07:44:43.665211 1 controller.go:244] error processing service default/nginx-deployment (will retry): failed to ensure load balancer: error retrieving load balancer rules: Get http://10.1.35.76:8080/client/api?apiKey=m8GvJ-HIcVY9XurofyAvOLn6mV7aAURljtgpCVH13b5O48ej_ewMNjNo32iKe64oeSuYrzI_gEgg7JCTo_UjPA&command=listLoadBalancerRules&keyword=aefa6bff63d27411fa5be2a94f7416df&listall=true&response=json&signature=AHQ1nikqf3Vi7ISV9zwO%2FuxgjgU%3D: dial tcp 10.1.35.76:8080: i/o timeout I1204 07:44:43.665448 1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"nginx-deployment", UID:"efa6bff6-3d27-411f-a5be-2a94f7416df7", APIVersion:"v1", ResourceVersion:"995877", FieldPath:""}): type: 'Warning' reason: 'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load balancer: error retrieving load balancer rules: Get http://10.1.35.76:8080/client/api?apiKey=m8GvJ-HIcVY9XurofyAvOLn6mV7aAURljtgpCVH13b5O48ej_ewMNjNo32iKe64oeSuYrzI_gEgg7JCTo_UjPA&command=listLoadBalancerRules&keyword=aefa6bff63d27411fa5be2a94f7416df&listall=true&response=json&signature=AHQ1nikqf3Vi7ISV9zwO%2FuxgjgU%3D: dial tcp 10.1.35.76:8080: i/o timeout E1204 07:45:39.310773 1 node_controller.go:245] Error getting node addresses for node "test-node-19377261815": error fetching node by provider ID: error retrieving node addresses: Get http://10.1.35.76:8080/client/api?apiKey=m8GvJ-HIcVY9XurofyAvOLn6mV7aAURljtgpCVH13b5O48ej_ewMNjNo32iKe64oeSuYrzI_gEgg7JCTo_UjPA&command=listVirtualMachines&id=&response=json&signature=547DXsoV%2BOnK945EvKkfLmLj4tU%3D: dial tcp 10.1.35.76:8080: i/o timeout, and error by node name: error retrieving node addresses: Get http://10.1.35.76:8080/client/api?apiKey=m8GvJ-HIcVY9XurofyAvOLn6mV7aAURljtgpCVH13b5O48ej_ewMNjNo32iKe64oeSuYrzI_gEgg7JCTo_UjPA&command=listVirtualMachines&name=test-node-19377261815&response=json&signature=1ONErKvq5OZxjpR26hzPmd8PLtY%3D: dial tcp 10.1.35.76:8080: i/o timeout E1204 07:46:20.106490 1 controller.go:719] failed to check if load balancer exists for service kube-system/nginx-lb: error retrieving load balancer rules: Get http://10.1.35.76:8080/client/api?apiKey=m8GvJ-HIcVY9XurofyAvOLn6mV7aAURljtgpCVH13b5O48ej_ewMNjNo32iKe64oeSuYrzI_gEgg7JCTo_UjPA&command=listLoadBalancerRules&keyword=ab9e4156d8f724ccfa61f379a8aaa0a6&listall=true&response=json&signature=MYL8q14GpEp5QxnRDg4nRXnkhQg%3D: dial tcp 10.1.35.76:8080: i/o timeout E1204 07:46:20.106727 1 controller.go:685] failed to update load balancer hosts for service kube-system/nginx-lb: error retrieving load balancer rules: Get http://10.1.35.76:8080/client/api?apiKey=m8GvJ-HIcVY9XurofyAvOLn6mV7aAURljtgpCVH13b5O48ej_ewMNjNo32iKe64oeSuYrzI_gEgg7JCTo_UjPA&command=listLoadBalancerRules&keyword=ab9e4156d8f724ccfa61f379a8aaa0a6&listall=true&response=json&signature=MYL8q14GpEp5QxnRDg4nRXnkhQg%3D: dial tcp 10.1.35.76:8080: i/o timeout I1204 07:46:20.107496 1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"kube-system", Name:"nginx-lb", UID:"b9e4156d-8f72-4ccf-a61f-379a8aaa0a63", APIVersion:"v1", ResourceVersion:"937", FieldPath:""}): type: 'Warning' reason: 'UpdateLoadBalancerFailed' Error updating load balancer with new hosts map[test-control-19377251026:{} test-node-19377261815:{}]: error retrieving load balancer rules: Get http://10.1.35.76:8080/client/api?apiKey=m8GvJ-HIcVY9XurofyAvOLn6mV7aAURljtgpCVH13b5O48ej_ewMNjNo32iKe64oeSuYrzI_gEgg7JCTo_UjPA&command=listLoadBalancerRules&keyword=ab9e4156d8f724ccfa61f379a8aaa0a6&listall=true&response=json&signature=MYL8q14GpEp5QxnRDg4nRXnkhQg%3D: dial tcp 10.1.35.76:8080: i/o timeout ``` So basically a user on a vmware is unable to access the Kubernetes application via the kubernetes loadbalancer service The workaround is expose the application via NodePort service Follow steps 1 to 6 11. Expose the application via node-port ``` kubectl expose deploy/nginx-deployment --port=80 --type=Nodeport NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 5d1h nginx-deployment2 NodePort 10.103.111.85 <none> 80:32014/TCP 4s ``` 12. Navigate to network and acquire a public IP  13. Allow the firewall port 80 on the public IP address  14. Add a loadbalcner rule mentioning the private node port  15. Add the kubernetes node  16. Access the application on the manually acquired public IP on port 80  I think for now we can document this workaround for CKS deployments on vmware and mention the loadbalancer service is not supported on vmware cc @rajujith @vishesh92 @weizhouapache Ref https://kubernetes.io/docs/tutorials/kubernetes-basics/expose/expose-intro/ https://kubernetes.io/docs/concepts/services-networking/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
