HeinzM opened a new issue, #241: URL: https://github.com/apache/cloudstack-terraform-provider/issues/241
Still getting an error related to [Issue 194](https://github.com/apache/cloudstack-terraform-provider/issues/194) [PR 198](https://github.com/apache/cloudstack-terraform-provider/pull/198) If I don't provide a project id to egress firewall the outcome is: except from terraform config: main.tf ``` terraform { required_providers { cloudstack = { source = "local/cloudstack/cloudstack" version = "0.6.0-rc2" } talos = { source = "siderolabs/talos" version = "0.9.0" } } } provider "cloudstack" { # Configuration options api_url = var.api_url api_key = var.api_key secret_key = var.secret_key } ``` network,tf ``` # Guest network used for K8s resource "cloudstack_network" "k8s_nw_01" { name = var.k8s_cluster_network_name cidr = "${local.network_cidr}" network_offering = "DefaultIsolatedNetworkOfferingWithSourceNatService" zone = var.zone project = var.project_id } # Egress firewall and rules for outside communication resource "cloudstack_egress_firewall" "default_egress_fw_01" { network_id = cloudstack_network.k8s_nw_01.id depends_on = [ cloudstack_instance.controller ] rule { cidr_list = ["${local.network_cidr}"] protocol = "tcp" ports = ["53", "80", "443"] } rule { cidr_list = ["${local.network_cidr}"] protocol = "udp" ports = ["53", "123"] } # TODO: Find out where to limit this rule to the destination ip of the Cloudstack API rule { cidr_list = ["${local.network_cidr}"] protocol = "tcp" ports = ["8443"] } } # SNAT ip address resource "cloudstack_ipaddress" "k8s_ips01" { network_id = cloudstack_network.k8s_nw_01.id project = var.project_id } # Load balancer for K8s API resource "cloudstack_loadbalancer_rule" "k8s_lb_k8s_api" { #depends_on = [ cloudstack_instance.controller ] project = var.project_id algorithm = "roundrobin" ip_address_id = cloudstack_ipaddress.k8s_ips01.id member_ids = [ cloudstack_instance.controller[0].id ] name = "lb-k8s-api" private_port = 6443 public_port = 6443 } # Loadbalancer for Talos API resource "cloudstack_loadbalancer_rule" "k8s_lb_talos_api" { #depends_on = [ cloudstack_instance.controller ] project = var.project_id algorithm = "roundrobin" ip_address_id = cloudstack_ipaddress.k8s_ips01.id member_ids = [ cloudstack_instance.controller[0].id ] name = "lb-talos-api" private_port = 50000 public_port = 50000 } # Firewall for accessing the SNAT ip address # TODO: set a valid range resource "cloudstack_firewall" "default_lbfw01" { ip_address_id = cloudstack_ipaddress.k8s_ips01.id depends_on = [ cloudstack_instance.controller ] rule { cidr_list = ["0.0.0.0/0"] protocol = "tcp" ports = ["6443", "50000"] } } ``` output apply: ``` module.k8s.talos_machine_secrets.talos: Creating... module.k8s.cloudstack_network.k8s_nw_01: Creating... module.k8s.cloudstack_network.k8s_nw_01: Creation complete after 1s [id=103adc96-9c51-446b-ac93-055f098364b3] module.k8s.cloudstack_ipaddress.k8s_ips01: Creating... module.k8s.talos_machine_secrets.talos: Creation complete after 2s [id=machine_secrets] module.k8s.data.talos_client_configuration.talos: Reading... module.k8s.data.talos_machine_configuration.worker: Reading... module.k8s.data.talos_client_configuration.talos: Read complete after 0s [id=test-cluster-mm] module.k8s.data.talos_machine_configuration.worker: Read complete after 0s [id=test-cluster-mm] module.k8s.cloudstack_ipaddress.k8s_ips01: Creation complete after 1s [id=f4f4162d-510c-4ff2-b630-c82eab3717a1] module.k8s.data.talos_machine_configuration.controller[0]: Reading... module.k8s.data.talos_machine_configuration.controller[0]: Read complete after 0s [id=test-cluster-mm] module.k8s.cloudstack_instance.controller[0]: Creating... module.k8s.cloudstack_instance.controller[0]: Still creating... [10s elapsed] module.k8s.cloudstack_instance.controller[0]: Still creating... [20s elapsed] module.k8s.cloudstack_instance.controller[0]: Still creating... [30s elapsed] module.k8s.cloudstack_instance.controller[0]: Still creating... [40s elapsed] module.k8s.cloudstack_instance.controller[0]: Creation complete after 46s [id=9dc18c37-d743-49f4-ae97-b19a952efe17] module.k8s.cloudstack_loadbalancer_rule.k8s_lb_k8s_api: Creating... module.k8s.cloudstack_loadbalancer_rule.k8s_lb_talos_api: Creating... module.k8s.cloudstack_instance.worker[1]: Creating... module.k8s.cloudstack_instance.worker[0]: Creating... module.k8s.cloudstack_firewall.default_lbfw01: Creating... module.k8s.cloudstack_egress_firewall.default_egress_fw_01: Creating... module.k8s.cloudstack_loadbalancer_rule.k8s_lb_k8s_api: Still creating... [10s elapsed] module.k8s.cloudstack_loadbalancer_rule.k8s_lb_talos_api: Still creating... [10s elapsed] module.k8s.cloudstack_instance.worker[1]: Still creating... [10s elapsed] module.k8s.cloudstack_instance.worker[0]: Still creating... [10s elapsed] module.k8s.cloudstack_egress_firewall.default_egress_fw_01: Still creating... [10s elapsed] module.k8s.cloudstack_loadbalancer_rule.k8s_lb_k8s_api: Creation complete after 10s [id=480f288c-06d2-4d9a-97c4-b5abf02fdf00] module.k8s.cloudstack_instance.worker[0]: Creation complete after 15s [id=1054386c-00e5-4a35-98b1-0ca3e57deb4e] module.k8s.cloudstack_loadbalancer_rule.k8s_lb_talos_api: Creation complete after 16s [id=f20ae740-6f1d-4a69-b7b1-821ac4222048] module.k8s.cloudstack_instance.worker[1]: Still creating... [20s elapsed] module.k8s.cloudstack_instance.worker[1]: Creation complete after 28s [id=82b4ddd2-e297-439c-94ff-cd1af1276b22] module.k8s.talos_machine_bootstrap.this: Creating... module.k8s.talos_machine_bootstrap.this: Creation complete after 0s [id=machine_bootstrap] module.k8s.talos_cluster_kubeconfig.kubeconfig: Creating... module.k8s.talos_cluster_kubeconfig.kubeconfig: Creation complete after 0s [id=test-cluster-mm] module.k8s.local_file.kubeconfig: Creating... module.k8s.local_file.kubeconfig: Creation complete after 0s [id=dd623c39ef9e7a1af82c8c2962ef66f27b9fb1df] ╷ │ Error: Provider produced inconsistent result after apply │ │ When applying changes to module.k8s.cloudstack_egress_firewall.default_egress_fw_01, provider "module.k8s.provider[\"local/cloudstack/cloudstack\"]" produced an unexpected new value: root object was present, but now absent. │ │ This is a bug in the provider, which should be reported in the provider's own issue tracker. ╵ ╷ │ Error: Provider produced inconsistent result after apply │ │ When applying changes to module.k8s.cloudstack_firewall.default_lbfw01, provider "module.k8s.provider[\"local/cloudstack/cloudstack\"]" produced an unexpected new value: root object was present, but now absent. │ │ This is a bug in the provider, which should be reported in the provider's own issue tracker. ``` output state list ``` module.k8s.data.talos_client_configuration.talos module.k8s.data.talos_machine_configuration.controller[0] module.k8s.data.talos_machine_configuration.worker module.k8s.cloudstack_instance.controller[0] module.k8s.cloudstack_instance.worker[0] module.k8s.cloudstack_instance.worker[1] module.k8s.cloudstack_ipaddress.k8s_ips01 module.k8s.cloudstack_loadbalancer_rule.k8s_lb_k8s_api module.k8s.cloudstack_loadbalancer_rule.k8s_lb_talos_api module.k8s.cloudstack_network.k8s_nw_01 module.k8s.local_file.kubeconfig module.k8s.talos_cluster_kubeconfig.kubeconfig module.k8s.talos_machine_bootstrap.this module.k8s.talos_machine_secrets.talos ``` cloudstack_egress_firewall is missing in state. with the project id: network.tf ``` locals { network_cidr = "${var.network_address}/${var.network_subnet}" gateway = cidrhost(local.network_cidr, 1) vip = cidrhost(local.network_cidr, 2) controller_nodes = [ for i in range(var.k8s_controller_count) : { name = "${var.k8s_cluster_name}-controller${format("%02d", i+1)}" ip = cidrhost("${local.network_cidr}", 3 + i) } ] worker_nodes = [ for i in range(var.k8s_worker_count) : { name = "${var.k8s_cluster_name}-worker${format("%02d", i+1)}" ip = cidrhost("${local.network_cidr}", 3 + var.k8s_controller_count + i) } ] cluster_endpoint = "https://${local.vip}:6443"; } resource "talos_machine_secrets" "talos" {} data "talos_machine_configuration" "controller" { count = var.k8s_controller_count cluster_name = var.k8s_cluster_name cluster_endpoint = local.cluster_endpoint machine_secrets = talos_machine_secrets.talos.machine_secrets machine_type = "controlplane" config_patches = [ yamlencode({ machine = { install = { disk = "/dev/sda" } env = { http_proxy = var.proxy_server https_proxy = var.proxy_server no_proxy = var.no_proxy } time = { servers = var.ntp_servers } kubelet = { extraArgs = { rotate-server-certificates = true } } network = { hostname = local.controller_nodes[count.index].name interfaces = [ { deviceSelector = { physical = true } addresses: [ "${local.controller_nodes[count.index].ip}/${var.network_subnet}" ] routes: [ { network = "0.0.0.0/0" gateway = local.gateway } ] vip = { ip = "${local.vip}" } } ] nameservers = var.dns_servers } } cluster = { network = { cni = { name = "none" } } proxy = { disabled = true } apiServer = { certSANs = ["${local.vip}", "${cloudstack_ipaddress.k8s_ips01.ip_address}"] } extraManifests = [ "https://raw.githubusercontent.com/alex1989hu/kubelet-serving-cert-approver/main/deploy/standalone-install.yaml";, "https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml"; ] } }) ] } data "talos_machine_configuration" "worker" { cluster_name = var.k8s_cluster_name cluster_endpoint = "https://${local.vip}:6443"; machine_secrets = talos_machine_secrets.talos.machine_secrets machine_type = "worker" config_patches = [ yamlencode({ machine = { install = { disk = "/dev/sda" } env = { http_proxy = var.proxy_server https_proxy = var.proxy_server no_proxy = var.no_proxy } time = { servers = var.ntp_servers } } cluster = { network = { cni = { name = "none" } } proxy = { disabled = true } } }) ] } data "talos_client_configuration" "talos" { cluster_name = var.k8s_cluster_name client_configuration = talos_machine_secrets.talos.client_configuration endpoints = [for node in local.controller_nodes : node.ip] nodes = [local.controller_nodes[0].ip] } resource "cloudstack_instance" "controller" { depends_on = [ ] count = var.k8s_controller_count project = var.project_id service_offering = var.service_offering_cp template = var.talos_image name = local.controller_nodes[count.index].name ip_address = local.controller_nodes[count.index].ip zone = var.zone cluster_id = var.cluster_ids[count.index % length(var.cluster_ids)] network_id = cloudstack_network.k8s_nw_01.id user_data = data.talos_machine_configuration.controller[count.index].machine_configuration expunge = true } resource "cloudstack_instance" "worker" { depends_on = [ cloudstack_instance.controller ] count = var.k8s_worker_count project = var.project_id service_offering = var.service_offering_worker template = var.talos_image name = local.worker_nodes[count.index].name ip_address = local.worker_nodes[count.index].ip zone = var.zone cluster_id = var.cluster_ids[count.index % length(var.cluster_ids)] network_id = cloudstack_network.k8s_nw_01.id user_data = data.talos_machine_configuration.worker.machine_configuration expunge = true root_disk_size = 16 } resource "talos_machine_bootstrap" "this" { depends_on = [ cloudstack_instance.controller, cloudstack_instance.worker, cloudstack_loadbalancer_rule.k8s_lb_talos_api ] endpoint = cloudstack_ipaddress.k8s_ips01.ip_address node = local.controller_nodes[0].ip client_configuration = talos_machine_secrets.talos.client_configuration } resource "talos_cluster_kubeconfig" "kubeconfig" { depends_on = [ talos_machine_bootstrap.this ] client_configuration = talos_machine_secrets.talos.client_configuration node = cloudstack_ipaddress.k8s_ips01.ip_address } locals { kubeconfig_modified = replace( talos_cluster_kubeconfig.kubeconfig.kubeconfig_raw, "https://${local.vip}:6443";, "https://${cloudstack_ipaddress.k8s_ips01.ip_address}:6443"; ) } resource "local_file" "kubeconfig" { depends_on = [ talos_cluster_kubeconfig.kubeconfig ] content = local.kubeconfig_modified filename = "./out/kubeconfig-${var.k8s_cluster_name}" } ``` output apply: ``` module.k8s.talos_machine_secrets.talos: Creating... module.k8s.cloudstack_network.k8s_nw_01: Creating... module.k8s.talos_machine_secrets.talos: Creation complete after 0s [id=machine_secrets] module.k8s.data.talos_client_configuration.talos: Reading... module.k8s.data.talos_machine_configuration.worker: Reading... module.k8s.data.talos_client_configuration.talos: Read complete after 0s [id=test-cluster-mm] module.k8s.data.talos_machine_configuration.worker: Read complete after 0s [id=test-cluster-mm] module.k8s.cloudstack_network.k8s_nw_01: Creation complete after 0s [id=eee99502-d2f3-43f4-a0ad-6a15e621c84b] module.k8s.cloudstack_ipaddress.k8s_ips01: Creating... module.k8s.cloudstack_ipaddress.k8s_ips01: Creation complete after 1s [id=f4f4162d-510c-4ff2-b630-c82eab3717a1] module.k8s.data.talos_machine_configuration.controller[0]: Reading... module.k8s.data.talos_machine_configuration.controller[0]: Read complete after 0s [id=test-cluster-mm] module.k8s.cloudstack_instance.controller[0]: Creating... module.k8s.cloudstack_instance.controller[0]: Still creating... [10s elapsed] module.k8s.cloudstack_instance.controller[0]: Still creating... [20s elapsed] module.k8s.cloudstack_instance.controller[0]: Still creating... [30s elapsed] module.k8s.cloudstack_instance.controller[0]: Still creating... [40s elapsed] module.k8s.cloudstack_instance.controller[0]: Still creating... [50s elapsed] module.k8s.cloudstack_instance.controller[0]: Creation complete after 56s [id=47e3b974-5f08-4dbf-bf0b-37f5b32cfde7] module.k8s.cloudstack_loadbalancer_rule.k8s_lb_talos_api: Creating... module.k8s.cloudstack_firewall.default_lbfw01: Creating... module.k8s.cloudstack_loadbalancer_rule.k8s_lb_k8s_api: Creating... module.k8s.cloudstack_instance.worker[0]: Creating... module.k8s.cloudstack_instance.worker[1]: Creating... module.k8s.cloudstack_egress_firewall.default_egress_fw_01: Creating... module.k8s.cloudstack_loadbalancer_rule.k8s_lb_talos_api: Creation complete after 6s [id=b47f9269-55f3-4189-a342-90e44e236190] module.k8s.cloudstack_loadbalancer_rule.k8s_lb_k8s_api: Still creating... [10s elapsed] module.k8s.cloudstack_instance.worker[0]: Still creating... [10s elapsed] module.k8s.cloudstack_instance.worker[1]: Still creating... [10s elapsed] module.k8s.cloudstack_egress_firewall.default_egress_fw_01: Still creating... [10s elapsed] module.k8s.cloudstack_egress_firewall.default_egress_fw_01: Creation complete after 13s [id=eee99502-d2f3-43f4-a0ad-6a15e621c84b] module.k8s.cloudstack_loadbalancer_rule.k8s_lb_k8s_api: Creation complete after 16s [id=82ca7dfc-a7a1-4846-b1b4-a9196db0d702] module.k8s.cloudstack_instance.worker[0]: Still creating... [20s elapsed] module.k8s.cloudstack_instance.worker[1]: Still creating... [20s elapsed] module.k8s.cloudstack_instance.worker[0]: Creation complete after 22s [id=39be1280-853b-4f22-9b72-f5c698dad14e] module.k8s.cloudstack_instance.worker[1]: Creation complete after 29s [id=fd08a259-8165-45f7-958a-1315d47b1efc] module.k8s.talos_machine_bootstrap.this: Creating... module.k8s.talos_machine_bootstrap.this: Still creating... [10s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [20s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [30s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [40s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [50s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [1m0s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [1m10s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [1m20s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [1m30s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [1m40s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [1m50s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [2m0s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [2m10s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [2m20s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [2m30s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [2m40s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [2m50s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [3m0s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [3m10s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [3m20s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [3m30s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [3m40s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [3m50s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [4m0s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [4m10s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [4m20s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [4m30s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [4m40s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [4m50s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [5m0s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [5m10s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [5m20s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [5m30s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [5m40s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [5m50s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [6m0s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [6m10s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [6m20s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [6m30s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [6m40s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [6m50s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [7m0s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [7m10s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [7m20s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [7m30s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [7m40s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [7m50s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [8m0s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [8m10s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [8m20s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [8m30s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [8m40s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [8m50s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [9m0s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [9m10s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [9m20s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [9m30s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [9m40s elapsed] module.k8s.talos_machine_bootstrap.this: Still creating... [9m50s elapsed] ╷ │ Error: Provider produced inconsistent result after apply │ │ When applying changes to module.k8s.cloudstack_firewall.default_lbfw01, provider "module.k8s.provider[\"local/cloudstack/cloudstack\"]" produced an unexpected new value: root object was present, but now absent. │ │ This is a bug in the provider, which should be reported in the provider's own issue tracker. ╵ ╷ │ Error: Error bootstrapping node │ │ with module.k8s.talos_machine_bootstrap.this, │ on modules/k8s/k8s-01.tf line 167, in resource "talos_machine_bootstrap" "this": │ 167: resource "talos_machine_bootstrap" "this" { │ │ rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate is valid for xxx.xxx.xxx.xxx ``` output state list: ``` module.k8s.data.talos_client_configuration.talos module.k8s.data.talos_machine_configuration.controller[0] module.k8s.data.talos_machine_configuration.worker module.k8s.cloudstack_egress_firewall.default_egress_fw_01 module.k8s.cloudstack_instance.controller[0] module.k8s.cloudstack_instance.worker[0] module.k8s.cloudstack_instance.worker[1] module.k8s.cloudstack_ipaddress.k8s_ips01 module.k8s.cloudstack_loadbalancer_rule.k8s_lb_k8s_api module.k8s.cloudstack_loadbalancer_rule.k8s_lb_talos_api module.k8s.cloudstack_network.k8s_nw_01 module.k8s.talos_machine_secrets.talos ``` With the project id provided to cloudstack_egress_firewall the egress firewall is available in the state, but bootstrapping the kubernetes cluster fails afterwards. We testet multiple times with and without projectid. Without the bootstrap is working flawlessly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
