Hi Sanket, Driver and executor logs are written to stdout by default, it can be configured using SPARK_HOME/conf/log4j.properties file. The file including the entire SPARK_HOME/conf is auto propogateded to all driver and executor container and mounted as volume.
Thanks On Mon, 9 Oct, 2023, 5:37 pm Agrawal, Sanket, <sankeagra...@deloitte.com.invalid> wrote: > Hi All, > > > > We are trying to send the spark logs using fluent-bit. We validated that > fluent-bit is able to move logs of all other pods except the > driver/executor pods. > > > > It would be great if someone can guide us where should I look for spark > logs in Spark on Kubernetes with client/cluster mode deployment. > > > > Thanks, > Sanket A. > > This message (including any attachments) contains confidential information > intended for a specific individual and purpose, and is protected by law. If > you are not the intended recipient, you should delete this message and any > disclosure, copying, or distribution of this message, or the taking of any > action based on it, by you is strictly prohibited. > > Deloitte refers to a Deloitte member firm, one of its related entities, or > Deloitte Touche Tohmatsu Limited ("DTTL"). Each Deloitte member firm is a > separate legal entity and a member of DTTL. DTTL does not provide services > to clients. Please see www.deloitte.com/about to learn more. > > v.E.1 >