This is an automated email from the ASF dual-hosted git repository.
aicam pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/texera.git
The following commit(s) were added to refs/heads/main by this push:
new 99348fb660 refactor(helm): unify ingress and envoy proxy as a single
gateway (#4191)
99348fb660 is described below
commit 99348fb6604717e670695280ff5f37bf529a1433
Author: ali risheh <[email protected]>
AuthorDate: Tue Feb 10 14:08:44 2026 -0800
refactor(helm): unify ingress and envoy proxy as a single gateway (#4191)
<!--
Thanks for sending a pull request (PR)! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines:
[Contributing to
Texera](https://github.com/apache/texera/blob/main/CONTRIBUTING.md)
2. Ensure you have added or run the appropriate tests for your PR
3. If the PR is work in progress, mark it a draft on GitHub.
4. Please write your PR title to summarize what this PR proposes, we
are following Conventional Commits style for PR titles as well.
5. Be sure to keep the PR description updated to reflect all changes.
-->
### What changes were proposed in this PR?
<!--
Please clarify what changes you are proposing. The purpose of this
section
is to outline the changes. Here are some tips for you:
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
3. If it is a refactoring, clarify what has been changed.
3. It would be helpful to include a before-and-after comparison using
screenshots or GIFs.
4. Please consider writing useful notes for better and faster reviews.
-->
This PR consolidates the cluster networking architecture by replacing
multiple disparate ingress/proxy solutions with a single, unified
**Envoy Gateway** using the Kubernetes Gateway API.
**Previously:**
- **Texera Ingress**: Handled by `ingress-nginx` controller (separate
Helm dependency).
- **MinIO Ingress**: Configured separately, often requiring its own
ingress status or port exposure.
- **CU Envoy**: A standalone, manually maintained Envoy deployment was
used to proxy traffic to Computing Units (CUs).
**Now (with Envoy Gateway):**
- **Unified Gateway**: A single `Gateway` resource (`texera-gateway`)
manages traffic for Texera Webserver, MinIO, and Computing Units.
- **Gateway API**: Uses standard `HTTPRoute` resources to define routing
rules (prefix matching, rewrites) instead of proprietary Ingress
annotations or custom config.
- **SSL/TLS Automation**: Integrated `cert-manager` with Envoy Gateway
to automatically provision and renew Let's Encrypt certificates for both
the main Texera domain and the MinIO subdomain.
**New Kubernetes Resources**:
* `bin/k8s/templates/gateway.yaml`: Defines the `Gateway` resource,
configuring listeners for HTTP, HTTPS, and MinIO. Handles TLS
termination using Let's Encrypt via cert-manager.
* `bin/k8s/templates/routes.yaml`: Defines `HTTPRoute` resources.
* **Static Routes**: Standard path-based routing for Texera services
(Webserver, API, etc.).
* **Dynamic Routes**: Captures regex paths for Computing Units and
delegates them to the dynamic backend.
* `bin/k8s/templates/backend.yaml`: Defines a `Backend` resource of type
`DynamicResolver`. This allows Envoy to route to targets defined
dynamically (e.g., by the ExtAuth service modifying headers) rather than
static Kubernetes services.
* `bin/k8s/templates/security-policy.yaml`: Defines the `SecurityPolicy`
that attaches to the dynamic routes. It configures the External
Authorization filter to point to the `access-control-service`.
* `bin/k8s/templates/eg-config-hook.yaml`: A **Helm Hook**
(pre-install/pre-upgrade) that automatically patches the Envoy Gateway
configuration to enable necessary features (`enableBackend`,
`enableEnvoyPatchPolicy`) which are disabled by default. It ensures the
environment is correctly configured without manual intervention.
### Any related issues, documentation, discussions?
<!--
Please use this section to link other resources if not mentioned
already.
1. If this PR fixes an issue, please include `Fixes #1234`, `Resolves
#1234`
or `Closes #1234`. If it is only related, simply mention the issue
number.
2. If there is design documentation, please add the link.
3. If there is a discussion in the mailing list, please add the link.
-->
Closes #4190
### How was this PR tested?
<!--
If tests were added, say they were added here. Or simply mention that if
the PR
is tested with existing test cases. Make sure to include/update test
cases that
check the changes thoroughly including negative and positive cases if
possible.
If it was tested in a way different from regular unit tests, please
clarify how
you tested step by step, ideally copy and paste-able, so that other
reviewers can
test and check, and descendants can verify in the future. If tests were
not added,
please describe why they were not added and/or why it was difficult to
add.
-->
Tested on the production RKE2 cluster:
We tested on http and https on both production server and local
environment.
### Was this PR authored or co-authored using generative AI tooling?
<!--
If generative AI tooling has been used in the process of authoring this
PR,
please include the phrase: 'Generated-by: ' followed by the name of the
tool
and its version. If no, write 'No'.
Please refer to the [ASF Generative Tooling
Guidance](https://www.apache.org/legal/generative-tooling.html) for
details.
-->
Generated-by: Antigravity
---
.../service/resource/AccessControlResource.scala | 11 +-
bin/k8s/Chart.yaml | 10 +-
.../access-control-service-deployment.yaml | 4 +
bin/k8s/templates/envoy-config.yaml | 147 ---------------------
bin/k8s/templates/envoy-deployment.yaml | 49 -------
.../{envoy-service.yaml => gateway-backend.yaml} | 18 +--
bin/k8s/templates/gateway-routes.yaml | 142 ++++++++++++++++++++
...y-service.yaml => gateway-security-policy.yaml} | 33 +++--
bin/k8s/templates/gateway.yaml | 81 ++++++++++++
bin/k8s/templates/ingress.yaml | 53 --------
bin/k8s/templates/minio-ingress.yaml | 65 ---------
bin/k8s/values.yaml | 126 +++++-------------
common/config/src/main/resources/kubernetes.conf | 5 +-
.../apache/texera/config/KubernetesConfig.scala | 1 +
14 files changed, 304 insertions(+), 441 deletions(-)
diff --git
a/access-control-service/src/main/scala/org/apache/texera/service/resource/AccessControlResource.scala
b/access-control-service/src/main/scala/org/apache/texera/service/resource/AccessControlResource.scala
index 1f2e2bc11f..0cd52f4919 100644
---
a/access-control-service/src/main/scala/org/apache/texera/service/resource/AccessControlResource.scala
+++
b/access-control-service/src/main/scala/org/apache/texera/service/resource/AccessControlResource.scala
@@ -26,7 +26,7 @@ import jakarta.ws.rs.{Consumes, GET, POST, Path, Produces}
import org.apache.texera.auth.JwtParser.parseToken
import org.apache.texera.auth.SessionUser
import org.apache.texera.auth.util.{ComputingUnitAccess, HeaderField}
-import org.apache.texera.config.{GuiConfig, LLMConfig}
+import org.apache.texera.config.{GuiConfig, KubernetesConfig, LLMConfig}
import org.apache.texera.dao.jooq.generated.enums.PrivilegeEnum
import java.net.URLDecoder
@@ -121,12 +121,21 @@ object AccessControlResource extends LazyLogging {
return Response.status(Response.Status.FORBIDDEN).build()
}
+ // Dynamic Routing Logic
+ val workflowComputingUnitPoolName = KubernetesConfig.computeUnitPoolName
+ val workflowComputingUnitPoolNamespace =
KubernetesConfig.computeUnitPoolNamespace
+ val workflowComputingUnitPoolPort = KubernetesConfig.computeUnitPortNumber
+
+ val targetHost =
+
s"computing-unit-$cuidInt.$workflowComputingUnitPoolName-svc.$workflowComputingUnitPoolNamespace.svc.cluster.local:$workflowComputingUnitPoolPort"
+
Response
.ok()
.header(HeaderField.UserComputingUnitAccess, cuAccess.toString)
.header(HeaderField.UserId, userSession.get().getUid.toString)
.header(HeaderField.UserName, userSession.get().getName)
.header(HeaderField.UserEmail, userSession.get().getEmail)
+ .header("Host", targetHost) // Envoy ExtAuth: Rewrite Host
.build()
}
diff --git a/bin/k8s/Chart.yaml b/bin/k8s/Chart.yaml
index ee57a2b842..1e6dcfbef8 100644
--- a/bin/k8s/Chart.yaml
+++ b/bin/k8s/Chart.yaml
@@ -42,11 +42,6 @@ appVersion: "1.16.0"
dependencies:
- - name: ingress-nginx
- version: 4.11.3
- repository: https://kubernetes.github.io/ingress-nginx
- condition: ingress-nginx.enabled
-
- name: postgresql
version: 16.5.6
repository: https://charts.bitnami.com/bitnami
@@ -59,6 +54,11 @@ dependencies:
version: 1.4.5
repository: https://charts.lakefs.io
+ - name: gateway-helm
+ version: 1.6.3
+ repository: oci://docker.io/envoyproxy
+ alias: envoy-gateway
+
- name: metrics-server
version: 3.12.2
repository: https://kubernetes-sigs.github.io/metrics-server/
diff --git a/bin/k8s/templates/access-control-service-deployment.yaml
b/bin/k8s/templates/access-control-service-deployment.yaml
index a332c65bd3..9da05bfd26 100644
--- a/bin/k8s/templates/access-control-service-deployment.yaml
+++ b/bin/k8s/templates/access-control-service-deployment.yaml
@@ -46,6 +46,10 @@ spec:
secretKeyRef:
name: {{ .Release.Name }}-postgresql
key: postgres-password
+ - name: KUBERNETES_COMPUTE_UNIT_POOL_NAME
+ value: {{ .Values.workflowComputingUnitPool.name }}
+ - name: KUBERNETES_COMPUTE_UNIT_POOL_NAMESPACE
+ value: {{ .Values.workflowComputingUnitPool.namespace }}
livenessProbe:
httpGet:
path: /api/healthcheck
diff --git a/bin/k8s/templates/envoy-config.yaml
b/bin/k8s/templates/envoy-config.yaml
deleted file mode 100644
index 2ba17e63f8..0000000000
--- a/bin/k8s/templates/envoy-config.yaml
+++ /dev/null
@@ -1,147 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-apiVersion: v1
-kind: ConfigMap
-metadata:
- name: envoy-config
- namespace: {{ .Release.Namespace }}
-data:
- envoy.yaml: |
- static_resources:
- listeners:
- - name: listener_0
- address:
- socket_address:
- address: 0.0.0.0
- port_value: 10000
- filter_chains:
- - filters:
- - name: envoy.filters.network.http_connection_manager
- typed_config:
- "@type":
type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
- stat_prefix: ingress_http
- upgrade_configs:
- - upgrade_type: websocket
- route_config:
- name: local_route
- virtual_hosts:
- - name: local_service
- domains: ["*"]
- routes:
- - match:
- prefix: "/wsapi"
- route:
- cluster: dynamic_service
- prefix_rewrite: "/wsapi"
- timeout: "0s" # disables timeout
- - match:
- safe_regex:
- google_re2: {}
- regex: "^/api/executions/\\d+/stats/\\d+$"
- route:
- cluster: dynamic_service
- timeout: "0s"
- - match:
- prefix: "/api/executions/result/export"
- route:
- cluster: dynamic_service
- prefix_rewrite: "/api/executions/result/export"
- timeout: "0s"
- http_filters:
- - name: envoy.filters.http.ext_authz
- typed_config:
- "@type":
type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
- with_request_body:
- # Max allowed by Envoy (uint32): 4,294,967,295
bytes (~4 GiB)
- max_request_bytes: 4294967295
- allow_partial_message: true # forward partial
body if larger than limit
- http_service:
- server_uri:
- uri: http://{{ .Release.Name }}-{{
.Values.accessControlService.name }}-svc.{{ .Release.Namespace
}}.svc.cluster.local
- cluster: access_control_service_cluster
- timeout: 5s
- path_prefix: /api/auth
- authorization_request:
- allowed_headers:
- patterns:
- - exact: "authorization"
- - exact: "x-envoy-original-path"
- authorization_response:
- allowed_upstream_headers:
- patterns:
- - exact: "x-user-cu-access"
- - exact: "x-user-id"
- - exact: "x-user-name"
- - exact: "x-user-email"
- failure_mode_allow: false
- - name: envoy.filters.http.lua
- typed_config:
- "@type":
type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
- inline_code: |
- function envoy_on_request(request_handle)
- local uri =
request_handle:headers():get(":path")
- local cuid = string.match(uri, "cuid=(%d+)")
- if cuid then
- local new_host = "computing-unit-" .. cuid
.. ".{{ .Values.workflowComputingUnitPool.name }}-svc.{{
.Values.workflowComputingUnitPool.namespace }}.svc.cluster.local:{{
.Values.workflowComputingUnitPool.service.port }}"
-
request_handle:headers():replace(":authority", new_host)
- end
- end
- - name: envoy.filters.http.dynamic_forward_proxy
- typed_config:
- "@type":
type.googleapis.com/envoy.extensions.filters.http.dynamic_forward_proxy.v3.FilterConfig
- dns_cache_config:
- name: dynamic_dns_cache
- dns_lookup_family: V4_ONLY
- dns_refresh_rate: 1s
- - name: envoy.filters.http.router
- typed_config:
- "@type":
type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
-
- access_log:
- - name: envoy.access_loggers.stdout
- typed_config:
- "@type":
type.googleapis.com/envoy.extensions.access_loggers.stream.v3.StdoutAccessLog
- log_format:
- text_format_source:
- inline_string: "[%START_TIME%] \"%REQ(:METHOD)%
%REQ(X-ENVOY-ORIGINAL-PATH?PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS%
%BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)%
\"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\"
\"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\"
AuthHeaders[user-id:\"%REQ(X-USER-ID)%\" email:\"%REQ(X-USER-EMAIL)%\"
cu-access:\"%REQ(X-USER-CU-ACCESS)%\"]\n"
-
- clusters:
- - name: dynamic_service
- connect_timeout: 0.25s
- lb_policy: CLUSTER_PROVIDED
- cluster_type:
- name: envoy.clusters.dynamic_forward_proxy
- typed_config:
- "@type":
type.googleapis.com/envoy.extensions.clusters.dynamic_forward_proxy.v3.ClusterConfig
- dns_cache_config:
- name: dynamic_dns_cache
- dns_lookup_family: V4_ONLY
- dns_refresh_rate: 1s
- - name: access_control_service_cluster # The Access Control Service
authorize requests and add user info headers
- connect_timeout: 0.25s
- type: LOGICAL_DNS
- lb_policy: round_robin
- load_assignment:
- cluster_name: access_control_service_cluster
- endpoints:
- - lb_endpoints:
- - endpoint:
- address:
- socket_address:
- address: {{ .Release.Name }}-{{
.Values.accessControlService.name }}-svc.{{ .Release.Namespace
}}.svc.cluster.local.
- port_value: {{
.Values.accessControlService.service.port }}
diff --git a/bin/k8s/templates/envoy-deployment.yaml
b/bin/k8s/templates/envoy-deployment.yaml
deleted file mode 100644
index 0e0522ee49..0000000000
--- a/bin/k8s/templates/envoy-deployment.yaml
+++ /dev/null
@@ -1,49 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-apiVersion: apps/v1
-kind: Deployment
-metadata:
- name: {{ .Release.Name }}-envoy-deployment
- namespace: {{ .Release.Namespace }}
-spec:
- replicas: {{ .Values.envoy.replicas | default 1 }}
- selector:
- matchLabels:
- app: envoy
- template:
- metadata:
- labels:
- app: envoy
- spec:
- containers:
- - name: envoy
- image: "{{ .Values.envoy.image.repository }}:{{
.Values.envoy.image.tag }}"
- ports:
- - containerPort: {{ .Values.envoy.port}}
- volumeMounts:
- - name: envoy-config
- mountPath: /etc/envoy
- readOnly: true
- args:
- - "-c"
- - "/etc/envoy/envoy.yaml" # Specify the path to the configuration
file
- - "--log-level debug" # Set level of logging
- volumes:
- - name: envoy-config
- configMap:
- name: envoy-config # Reference the ConfigMap created earlier
\ No newline at end of file
diff --git a/bin/k8s/templates/envoy-service.yaml
b/bin/k8s/templates/gateway-backend.yaml
similarity index 65%
copy from bin/k8s/templates/envoy-service.yaml
copy to bin/k8s/templates/gateway-backend.yaml
index b936a7702b..2db3cdc3ae 100644
--- a/bin/k8s/templates/envoy-service.yaml
+++ b/bin/k8s/templates/gateway-backend.yaml
@@ -15,20 +15,10 @@
# specific language governing permissions and limitations
# under the License.
-apiVersion: v1
-kind: Service
+apiVersion: gateway.envoyproxy.io/v1alpha1
+kind: Backend
metadata:
- name: envoy-svc
+ name: {{ .Release.Name }}-dynamic-backend
namespace: {{ .Release.Namespace }}
spec:
- type: {{ .Values.envoy.service.type }}
- selector:
- app: envoy
- ports:
- - protocol: TCP
- port: {{ .Values.envoy.service.port }}
- targetPort: {{ .Values.envoy.service.port }}
- # if service type is set to NodePort, include nodePort attribute
- {{- if eq .Values.envoy.service.type "NodePort" }}
- nodePort: {{ .Values.envoy.service.nodePort }}
- {{- end }}
\ No newline at end of file
+ type: DynamicResolver
diff --git a/bin/k8s/templates/gateway-routes.yaml
b/bin/k8s/templates/gateway-routes.yaml
new file mode 100644
index 0000000000..55dc40f581
--- /dev/null
+++ b/bin/k8s/templates/gateway-routes.yaml
@@ -0,0 +1,142 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+ name: {{ .Release.Name }}-static-routes
+ namespace: {{ .Release.Namespace }}
+spec:
+ parentRefs:
+ - name: {{ .Release.Name }}-gateway
+ {{- if and .Values.gatewayConfig .Values.gatewayConfig.hostname }}
+ hostnames:
+ - {{ .Values.gatewayConfig.hostname }}
+ {{- end }}
+ rules:
+ - matches:
+ - path:
+ type: PathPrefix
+ value: /api/computing-unit
+ backendRefs:
+ - name: workflow-computing-unit-manager-svc
+ port: 8888
+ - matches:
+ - path:
+ type: PathPrefix
+ value: /api/compile
+ backendRefs:
+ - name: workflow-compiling-service-svc
+ port: 9090
+ - matches:
+ - path:
+ type: PathPrefix
+ value: /api/dataset
+ - path:
+ type: PathPrefix
+ value: /api/access/dataset
+ backendRefs:
+ - name: file-service-svc
+ port: 9092
+ - matches:
+ - path:
+ type: PathPrefix
+ value: /api/access/computing-unit
+ backendRefs:
+ - name: workflow-computing-unit-manager-svc
+ port: 8888
+ - matches:
+ - path:
+ type: PathPrefix
+ value: /api/config
+ backendRefs:
+ - name: config-service-svc
+ port: 9094
+ - matches:
+ - path:
+ type: PathPrefix
+ value: /rtc
+ backendRefs:
+ - name: y-websocket-server-svc
+ port: 1234
+ - matches:
+ - path:
+ type: PathPrefix
+ value: /python-language-server
+ backendRefs:
+ - name: python-language-server-svc
+ port: 3000
+ - matches:
+ - path:
+ type: PathPrefix
+ value: /api
+ - path:
+ type: PathPrefix
+ value: /
+ backendRefs:
+ - name: webserver-svc
+ port: 8080
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+ name: {{ .Release.Name }}-dynamic-routes
+ namespace: {{ .Release.Namespace }}
+spec:
+ parentRefs:
+ - name: {{ .Release.Name }}-gateway
+ {{- if and .Values.gatewayConfig .Values.gatewayConfig.hostname }}
+ hostnames:
+ - {{ .Values.gatewayConfig.hostname }}
+ {{- end }}
+ rules:
+ - matches:
+ - path:
+ type: PathPrefix
+ value: /wsapi
+ - path:
+ type: RegularExpression
+ value: "^/api/executions/\\d+/stats/\\d+$"
+ - path:
+ type: PathPrefix
+ value: /api/executions/result/export
+ backendRefs:
+ - group: gateway.envoyproxy.io
+ kind: Backend
+ name: texera-dynamic-backend
+---
+# MinIO Route
+{{- if .Values.minio.gateway.enabled }}
+apiVersion: gateway.networking.k8s.io/v1
+kind: HTTPRoute
+metadata:
+ name: texera-minio-route
+ namespace: {{ .Release.Namespace }}
+spec:
+ parentRefs:
+ - name: {{ .Release.Name }}-gateway
+ hostnames:
+ - {{ .Values.minio.gateway.hostname }}
+ rules:
+ - matches:
+ - path:
+ type: PathPrefix
+ value: /
+ backendRefs:
+ - name: {{ .Release.Name }}-minio
+ port: 9000
+{{- end }}
diff --git a/bin/k8s/templates/envoy-service.yaml
b/bin/k8s/templates/gateway-security-policy.yaml
similarity index 60%
rename from bin/k8s/templates/envoy-service.yaml
rename to bin/k8s/templates/gateway-security-policy.yaml
index b936a7702b..ad2f5684e5 100644
--- a/bin/k8s/templates/envoy-service.yaml
+++ b/bin/k8s/templates/gateway-security-policy.yaml
@@ -15,20 +15,25 @@
# specific language governing permissions and limitations
# under the License.
-apiVersion: v1
-kind: Service
+apiVersion: gateway.envoyproxy.io/v1alpha1
+kind: SecurityPolicy
metadata:
- name: envoy-svc
+ name: {{ .Release.Name }}-ext-auth
namespace: {{ .Release.Namespace }}
spec:
- type: {{ .Values.envoy.service.type }}
- selector:
- app: envoy
- ports:
- - protocol: TCP
- port: {{ .Values.envoy.service.port }}
- targetPort: {{ .Values.envoy.service.port }}
- # if service type is set to NodePort, include nodePort attribute
- {{- if eq .Values.envoy.service.type "NodePort" }}
- nodePort: {{ .Values.envoy.service.nodePort }}
- {{- end }}
\ No newline at end of file
+ targetRefs:
+ - group: gateway.networking.k8s.io
+ kind: HTTPRoute
+ name: {{ .Release.Name }}-dynamic-routes
+ extAuth:
+ http:
+ backendRefs:
+ - name: {{ .Release.Name }}-{{ .Values.accessControlService.name }}-svc
+ port: {{ .Values.accessControlService.service.port }}
+ path: /api/auth
+ headersToBackend:
+ - x-user-cu-access
+ - x-user-id
+ - x-user-name
+ - x-user-email
+ - Host
diff --git a/bin/k8s/templates/gateway.yaml b/bin/k8s/templates/gateway.yaml
new file mode 100644
index 0000000000..868439a673
--- /dev/null
+++ b/bin/k8s/templates/gateway.yaml
@@ -0,0 +1,81 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+apiVersion: gateway.networking.k8s.io/v1
+kind: GatewayClass
+metadata:
+ name: eg
+spec:
+ controllerName: gateway.envoyproxy.io/gatewayclass-controller
+---
+apiVersion: gateway.networking.k8s.io/v1
+kind: Gateway
+metadata:
+ name: {{ .Release.Name }}-gateway
+ annotations:
+ {{- if and .Values.gatewayConfig .Values.gatewayConfig.issuer }}
+ {{- if eq (default "Issuer" .Values.gatewayConfig.issuerKind)
"ClusterIssuer" }}
+ cert-manager.io/cluster-issuer: {{ .Values.gatewayConfig.issuer }}
+ {{- else }}
+ cert-manager.io/issuer: {{ .Values.gatewayConfig.issuer }}
+ {{- end }}
+ {{- end }}
+spec:
+ gatewayClassName: eg
+ listeners:
+ - name: http
+ protocol: HTTP
+ port: 80
+ {{- if and .Values.gatewayConfig .Values.gatewayConfig.hostname }}
+ hostname: {{ .Values.gatewayConfig.hostname }}
+ {{- end }}
+ allowedRoutes:
+ namespaces:
+ from: Same
+ - name: https
+ protocol: HTTPS
+ port: 443
+ {{- if and .Values.gatewayConfig .Values.gatewayConfig.hostname }}
+ hostname: {{ .Values.gatewayConfig.hostname }}
+ {{- end }}
+ allowedRoutes:
+ namespaces:
+ from: Same
+ tls:
+ mode: Terminate
+ certificateRefs:
+ - name: {{ (index .Values "gatewayConfig" "tlsSecretName") | default
(printf "%s-cert" .Release.Name) }}
+ {{- if .Values.minio.gateway.enabled }}
+ - name: minio-http
+ protocol: HTTP
+ port: 80
+ hostname: {{ .Values.minio.gateway.hostname }}
+ allowedRoutes:
+ namespaces:
+ from: Same
+ - name: minio-https
+ protocol: HTTPS
+ port: 443
+ hostname: {{ .Values.minio.gateway.hostname }}
+ allowedRoutes:
+ namespaces:
+ from: Same
+ tls:
+ mode: Terminate
+ certificateRefs:
+ - name: {{ .Values.minio.gateway.tlsSecretName | default (printf
"%s-minio-tls" .Release.Name) }}
+ {{- end }}
diff --git a/bin/k8s/templates/ingress.yaml b/bin/k8s/templates/ingress.yaml
deleted file mode 100644
index 43c5bd3fc1..0000000000
--- a/bin/k8s/templates/ingress.yaml
+++ /dev/null
@@ -1,53 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-{{- if .Values.ingressPaths.enabled }}
-apiVersion: networking.k8s.io/v1
-kind: Ingress
-metadata:
- name: {{ .Release.Name }}-ingress
- namespace: {{ .Release.Namespace }}
- annotations:
- nginx.ingress.kubernetes.io/use-regex: "true"
- {{- if or .Values.ingressPaths.tlsSecretName .Values.ingressPaths.issuer }}
- nginx.ingress.kubernetes.io/ssl-redirect: {{ or
.Values.ingressPaths.tlsSecretName .Values.ingressPaths.issuer | quote}}
- {{- end }}
- {{- if .Values.ingressPaths.issuer }}
- cert-manager.io/issuer: {{ .Values.ingressPaths.issuer }}
- {{- end }}
-spec:
- ingressClassName: {{ .Values.ingressPaths.ingressClassName }}
- {{- if or .Values.ingressPaths.tlsSecretName .Values.ingressPaths.issuer }}
- tls:
- - hosts:
- - {{ .Values.ingressPaths.hostname }}
- secretName: {{ .Values.ingressPaths.tlsSecretName | default (printf
"%s-tls" .Release.Name) }}
- {{- end }}
- rules:
- - host: {{ .Values.ingressPaths.hostname }}
- http:
- paths:
- {{- range .Values.ingressPaths.paths }}
- - path: {{ .path }}
- pathType: {{ .pathType | default "Prefix" }}
- backend:
- service:
- name: {{ .serviceName }}
- port:
- number: {{ .servicePort }}
- {{- end }}
-{{- end }}
\ No newline at end of file
diff --git a/bin/k8s/templates/minio-ingress.yaml
b/bin/k8s/templates/minio-ingress.yaml
deleted file mode 100644
index 24bfab2cd7..0000000000
--- a/bin/k8s/templates/minio-ingress.yaml
+++ /dev/null
@@ -1,65 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-{{- if .Values.minio.customIngress.enabled }}
-apiVersion: networking.k8s.io/v1
-kind: Ingress
-metadata:
- name: texera-minio-api-ingress
- namespace: {{ .Release.Namespace }}
- annotations:
- kubernetes.io/ingress.class: {{
.Values.minio.customIngress.ingressClassName }}
- {{- if .Values.minio.customIngress.issuer }}
- cert-manager.io/issuer: {{ .Values.minio.customIngress.issuer }}
- {{- end }}
- nginx.ingress.kubernetes.io/enable-cors: "true"
- nginx.ingress.kubernetes.io/cors-allow-origin: "{{
.Values.minio.customIngress.texeraHostname }}"
- nginx.ingress.kubernetes.io/cors-allow-methods: "GET, PUT, POST, DELETE,
OPTIONS"
- nginx.ingress.kubernetes.io/cors-allow-headers: "*"
- nginx.ingress.kubernetes.io/cors-expose-headers: "ETag,
x-amz-meta-custom-header"
- nginx.ingress.kubernetes.io/cors-max-age: "86400"
- nginx.ingress.kubernetes.io/proxy-body-size: "0"
- nginx.ingress.kubernetes.io/proxy-buffering: "off"
- nginx.ingress.kubernetes.io/proxy-read-timeout: "300"
- nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
- nginx.ingress.kubernetes.io/proxy-send-timeout: "300"
- nginx.ingress.kubernetes.io/use-regex: "true"
- nginx.ingress.kubernetes.io/x-forwarded-proto: "https"
- nginx.ingress.kubernetes.io/backend-protocol: "HTTP"
- {{- if or .Values.ingressPaths.tlsSecretName .Values.ingressPaths.issuer }}
- nginx.ingress.kubernetes.io/ssl-redirect: "true"
- {{- end }}
-spec:
- ingressClassName: {{ .Values.minio.customIngress.ingressClassName }}
- {{- if or .Values.ingressPaths.tlsSecretName .Values.ingressPaths.issuer }}
- tls:
- - hosts:
- - {{ .Values.minio.customIngress.minioHostname }}
- secretName: {{ .Values.ingressPaths.tlsSecretName | default (printf
"%s-minio-tls" .Release.Name) }}
- {{- end }}
- rules:
- - host: {{ .Values.minio.customIngress.minioHostname }}
- http:
- paths:
- - path: /
- pathType: Prefix
- backend:
- service:
- name: {{ .Release.Name }}-minio
- port:
- number: 9000
-{{- end }}
diff --git a/bin/k8s/values.yaml b/bin/k8s/values.yaml
index 11594422d3..ee0de26e20 100644
--- a/bin/k8s/values.yaml
+++ b/bin/k8s/values.yaml
@@ -52,7 +52,6 @@ postgresql:
size: 10Gi
storageClass: local-path
existingClaim: "postgresql-data-pvc"
-
initdb:
scriptsConfigMap: "postgresql-init-script"
@@ -61,12 +60,10 @@ minio:
image:
repository: bitnamilegacy/minio
tag: 2025.3.12-debian-12-r0
- customIngress:
+ gateway:
enabled: false
- ingressClassName: "" # e.g., "nginx"
- texeraHostname: "" # the url for the texera WITH http or https, e.g.,
"https://texera.example.com";
- minioHostname: "" # the url for the minio
- issuer: "" # e.g., "letsencrypt-prod"
+ hostname: "" # the url for the minio, e.g. "minio.example.com"
+ tlsSecretName: "" # e.g. "minio-tls-secret"
auth:
rootUser: texera_minio
rootPassword: password
@@ -168,19 +165,6 @@ accessControlService:
type: ClusterIP
port: 9096
-# Configs of the envoy proxy, used to routerequests to the computing units
-envoy:
- replicas: 1
- image:
- repository: envoyproxy/envoy
- tag: v1.31-latest
- port:
- 10000
- debug: false
- service:
- type: ClusterIP
- port: 10000
-
# headless service for the access of computing units
workflowComputingUnitPool:
createNamespaces: true
@@ -254,29 +238,6 @@ texeraEnvVars:
- name: USER_SYS_DOMAIN
value: ""
-# Ingress dependency configs
-ingress-nginx:
- enabled: true # set to true if nginx is not installed, should be false in
production
- controller:
- replicaCount: 1
- service:
- type: NodePort
- nodePorts:
- http: 30080
- ingressClassResource:
- name: nginx
- enabled: true
- resources:
- limits:
- cpu: 100m
- memory: 128Mi
- requests:
- cpu: 100m
- memory: 128Mi
- rbac:
- create: true
-
-
yWebsocketServer:
name: y-websocket-server
replicaCount: 1
@@ -310,54 +271,35 @@ metrics-server:
create: true
priorityClassName: system-cluster-critical
-# Custom Ingress resource configs
-ingressPaths:
- enabled: true
- ingressClassName: nginx # Set to "alb" when deploying on AWS
+gatewayConfig:
+ # Routes are available at bin/k8s/templates/gateway-routes.yaml
+
+ # The hostname for the Gateway listener (HTTP/HTTPS).
+ # e.g., "texera.example.com"
hostname: ""
- # Optional TLS secret (manually created)
- tlsSecretName: "" # e.g., "texera-tls"
- # Optional Issuer name for cert-manager
- issuer: "" # e.g., "letsencrypt-prod"
- paths:
- - path: /api/computing-unit
- serviceName: workflow-computing-unit-manager-svc
- servicePort: 8888
- - path: /api/compile
- serviceName: workflow-compiling-service-svc
- servicePort: 9090
- - path: /api/dataset
- serviceName: file-service-svc
- servicePort: 9092
- - path: /api/access/dataset
- serviceName: file-service-svc
- servicePort: 9092
- - path: /api/access/computing-unit
- serviceName: workflow-computing-unit-manager-svc
- servicePort: 8888
- - path: /api/config
- serviceName: config-service-svc
- servicePort: 9094
- - path: /wsapi/workflow-websocket
- serviceName: envoy-svc
- servicePort: 10000
- - path: /api/executions/*/stats/*
- pathType: ImplementationSpecific
- serviceName: envoy-svc
- servicePort: 10000
- - path: /api/executions/result/export/*
- pathType: ImplementationSpecific
- serviceName: envoy-svc
- servicePort: 10000
- - path: /api
- serviceName: webserver-svc
- servicePort: 8080
- - path: /rtc
- serviceName: y-websocket-server-svc
- servicePort: 1234
- - path: /python-language-server
- serviceName: python-language-server-svc
- servicePort: 3000
- - path: /
- serviceName: webserver-svc
- servicePort: 8080
+
+ # The name of the cert-manager Issuer or ClusterIssuer to use for obtaining
certificates.
+ # This requires cert-manager to be installed in the cluster.
+ # You can find available ClusterIssuers with: `kubectl get clusterissuers`
+ # You can find available Issuers with: `kubectl get issuers -A`
+ # e.g., "letsencrypt-prod"
+ issuer: ""
+
+ # The Kind of the issuer specified above. Can be "Issuer" or "ClusterIssuer".
+ # If you found it via `kubectl get clusterissuers`, use "ClusterIssuer".
+ # If you found it via `kubectl get issuers`, use "Issuer".
+ # defaults to "Issuer" if not specified.
+ issuerKind: "Issuer"
+
+ # The name of the Secret where the signed certificate should be stored.
+ # If empty, it defaults to "{{ .Release.Name }}-cert".
+ # e.g., "texera-tls"
+ tlsSecretName: ""
+
+# Envoy Gateway Configuration
+envoy-gateway:
+ config:
+ envoyGateway:
+ extensionApis:
+ enableBackend: true
+ enableEnvoyPatchPolicy: true
diff --git a/common/config/src/main/resources/kubernetes.conf
b/common/config/src/main/resources/kubernetes.conf
index f40fe55ea2..e85924e570 100644
--- a/common/config/src/main/resources/kubernetes.conf
+++ b/common/config/src/main/resources/kubernetes.conf
@@ -20,7 +20,10 @@ kubernetes {
enabled = false
enabled = ${?KUBERNETES_COMPUTING_UNIT_ENABLED}
- compute-unit-pool-namespace = "workflow-computing-unit-pool"
+ compute-unit-pool-name = "texera-workflow-computing-unit"
+ compute-unit-pool-name = ${?KUBERNETES_COMPUTE_UNIT_POOL_NAME}
+
+ compute-unit-pool-namespace = "texera-workflow-computing-unit-pool"
compute-unit-pool-namespace = ${?KUBERNETES_COMPUTE_UNIT_POOL_NAMESPACE}
compute-unit-service-name = "workflow-computing-unit-svc"
diff --git
a/common/config/src/main/scala/org/apache/texera/config/KubernetesConfig.scala
b/common/config/src/main/scala/org/apache/texera/config/KubernetesConfig.scala
index 7f8763e894..7f9a730fe8 100644
---
a/common/config/src/main/scala/org/apache/texera/config/KubernetesConfig.scala
+++
b/common/config/src/main/scala/org/apache/texera/config/KubernetesConfig.scala
@@ -28,6 +28,7 @@ object KubernetesConfig {
// Access the Kubernetes settings with environment variable fallback
val computeUnitServiceName: String =
conf.getString("kubernetes.compute-unit-service-name")
+ val computeUnitPoolName: String =
conf.getString("kubernetes.compute-unit-pool-name")
val computeUnitPoolNamespace: String =
conf.getString("kubernetes.compute-unit-pool-namespace")
val computeUnitImageName: String = conf.getString("kubernetes.image-name")
val computingUnitImagePullPolicy: String =
conf.getString("kubernetes.image-pull-policy")
