We have a similar setup (EKS/S3) and use promtail to collect pod logs to loki.
We haven't tried to get the history UI log links working. Instead we link to both the history server and logs from the same job/cluster overview dashboards in grafana. On Wed, Oct 23, 2024 at 3:36 PM karan alang <karan.al...@gmail.com> wrote: > Hello All, > I have kubeflow spark operator installed on GKE (in namespace - so350), as > well as Spark History Server installed on GKE in namespace shs-350. > The spark job is launched in a separate namespaces - spark-apps. > > When I launch the spark job, it runs fine and I'm able to see the job > details in the Spark History server UI. > The Spark History Server is configured to have the events stored in a GCP > storage bucket, where I see the event logs. > > While the events are getting stored, the worker & executor logs are not > getting stored in the storage bucket and hence not showing up in the > History Server UI. > i.e. the stderr/stdout links are not enabled on the Spark History Server > UI. > > How do I enable storing the worker/executor logs on a GCP bucket ? > Do I need to install fluentbit or fluentd to collect the logs from k8s > pods & store in the storage bucket ? > > > Any inputs on this ? > > tia! > > Pls note : to enable the spark events to flow into GCP storage bucket, CM > is created as shown below to update the spark-defaults.conf > > apiVersion: v1 > kind: ConfigMap > metadata: > name: spark-history-server-conf > namespace: shs-350 > data: > spark-defaults.conf: | > spark.history.fs.logDirectory=gs://<storage-bucket>/spark-events > spark.hadoop.google.cloud.auth.service.account.enable=true > > spark.hadoop.google.cloud.auth.service.account.json.keyfile=/etc/secrets/spark-gcs-key.json > > > > In the spark job yaml, following configuration is done : > > "spark.eventLog.enabled": "true" > "spark.eventLog.dir": "gs://<storage-bucket>/spark-events" >
