Spark should be using the gateway's configuration. Unless you're launching the application from a different node, if the setting is there, Spark should be using it.
You can also look in the UI's environment page to see the configuration that the app is using. On Mon, Mar 26, 2018 at 10:10 AM, Fawze Abujaber <fawz...@gmail.com> wrote: > I see this configuration only on the spark gateway server, and my spark is > running on Yarn, so I think I missing something ... > > I’m using cloudera manager to set this parameter, maybe I need to add this > parameter in other configuration > > On Mon, 26 Mar 2018 at 20:05 Marcelo Vanzin <van...@cloudera.com> wrote: >> >> If the spark-defaults.conf file in the machine where you're starting >> the Spark app has that config, then that's all that should be needed. >> >> On Mon, Mar 26, 2018 at 10:02 AM, Fawze Abujaber <fawz...@gmail.com> >> wrote: >> > Thanks Marcelo, >> > >> > Yes I was was expecting to see the new apps compressed but I don’t , do >> > I >> > need to perform restart to spark or Yarn? >> > >> > On Mon, 26 Mar 2018 at 19:53 Marcelo Vanzin <van...@cloudera.com> wrote: >> >> >> >> Log compression is a client setting. Doing that will make new apps >> >> write event logs in compressed format. >> >> >> >> The SHS doesn't compress existing logs. >> >> >> >> On Mon, Mar 26, 2018 at 9:17 AM, Fawze Abujaber <fawz...@gmail.com> >> >> wrote: >> >> > Hi All, >> >> > >> >> > I'm trying to compress the logs at SPark history server, i added >> >> > spark.eventLog.compress=true to spark-defaults.conf to spark Spark >> >> > Client >> >> > Advanced Configuration Snippet (Safety Valve) for >> >> > spark-conf/spark-defaults.conf >> >> > >> >> > which i see applied only to the spark gateway servers spark conf. >> >> > >> >> > What i missing to get this working ? >> >> >> >> >> >> >> >> -- >> >> Marcelo >> >> >> >> -- >> Marcelo -- Marcelo --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
