Aditya Vishwakarma created AIRFLOW-3545:
-------------------------------------------
Summary: Can't use Prometheus or other pull based instrumentation
systems to monitor Tasks launched on Kubernetes
Key: AIRFLOW-3545
URL: https://issues.apache.org/jira/browse/AIRFLOW-3545
Project: Apache Airflow
Issue Type: Bug
Reporter: Aditya Vishwakarma
Prometheus, which is a common way to instrument services on Kubernetes, uses a
pull based mechanism to fetch metrics. Which generally involves a service
exposing a `/metrics` endpoints. This endpoint is scraped every 30 secs by
prometheus to collect metrics.
This requires a port to be specified in the generated Pod config. Something
like below.
{code:java}
// Sample Pod Spec
apiVersion: v1
kind: Job
metadata:
name: batch-job
spec:
ports:
- name: metrics
port: 9091 # port to fetch metrics from
protocol: TCP
targetPort: 9091
{code}
Currently KubernetesPodOperator doesn't have any options to open ports like
this.
Is it possible to have an option to do this?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
