Hmm. Can you set the permissions of "/apps/spark/historyserver/logs" to 3777? I'm not sure HDFS respects the group id bit, but it's worth a try. (BTW that would only affect newly created log directories.)
On Thu, Jan 8, 2015 at 1:22 AM, <[email protected]> wrote: > Hi Vanzin, > > I am using the MapR distribution of Hadoop. The history server logs are > created by a job with the permissions: > > drwxrwx--- - <myusername> <mygroup> 2 2015-01-08 09:14 > /apps/spark/historyserver/logs/spark-1420708455212 > > However, the permissions of the higher directories are mapr:mapr and the user > that runs Spark in our case is a unix ID called mapr (in the mapr group). > Therefore, this can't read my job event logs as shown above. > > > Thanks, > Michael > > > -----Original Message----- > From: Marcelo Vanzin [mailto:[email protected]] > Sent: 07 January 2015 18:10 > To: England, Michael (IT/UK) > Cc: [email protected] > Subject: Re: Spark History Server can't read event logs > > The Spark code generates the log directory with "770" permissions. On top of > that you need to make sure of two things: > > - all directories up to /apps/spark/historyserver/logs/ are readable by the > user running the history server > - the user running the history server belongs to the group that owns > /apps/spark/historyserver/logs/ > > I think the code could be more explicitly about setting the group of the > generated log directories and files, but if you follow the two rules above > things should work. Also, I recommend setting /apps/spark/historyserver/logs/ > itself to "1777" so that any user can generate logs, but only the owner (or a > superuser) can delete them. > > > > On Wed, Jan 7, 2015 at 7:45 AM, <[email protected]> wrote: >> Hi, >> >> >> >> When I run jobs and save the event logs, they are saved with the >> permissions of the unix user and group that ran the spark job. The >> history server is run as a service account and therefore can’t read the >> files: >> >> >> >> Extract from the History server logs: >> >> >> >> 2015-01-07 15:37:24,3021 ERROR Client >> fs/client/fileclient/cc/client.cc:1009 >> Thread: 1183 User does not have access to open file >> /apps/spark/historyserver/logs/spark-1420644521194 >> >> 15/01/07 15:37:24 ERROR ReplayListenerBus: Exception in parsing Spark >> event log >> /apps/spark/historyserver/logs/spark-1420644521194/EVENT_LOG_1 >> >> org.apache.hadoop.security.AccessControlException: Open failed for file: >> /apps/spark/historyserver/logs/spark-1420644521194/EVENT_LOG_1, error: >> Permission denied (13) >> >> >> >> Is there a setting which I can change that allows the files to be >> world readable or at least by the account running the history server? >> Currently, the job appears in the History Sever UI but only states ‘<Not >> Started>’. >> >> >> >> Thanks, >> >> Michael >> >> >> This e-mail (including any attachments) is private and confidential, >> may contain proprietary or privileged information and is intended for >> the named >> recipient(s) only. Unintended recipients are strictly prohibited from >> taking action on the basis of information in this e-mail and must >> contact the sender immediately, delete this e-mail (and all >> attachments) and destroy any hard copies. Nomura will not accept >> responsibility or liability for the accuracy or completeness of, or >> the presence of any virus or disabling code in, this e-mail. If >> verification is sought please request a hard copy. Any reference to >> the terms of executed transactions should be treated as preliminary only and >> subject to formal written confirmation by Nomura. >> Nomura reserves the right to retain, monitor and intercept e-mail >> communications through its networks (subject to and in accordance with >> applicable laws). No confidentiality or privilege is waived or lost by >> Nomura by any mistransmission of this e-mail. Any reference to >> "Nomura" is a reference to any entity in the Nomura Holdings, Inc. >> group. Please read our Electronic Communications Legal Notice which forms >> part of this e-mail: >> http://www.Nomura.com/email_disclaimer.htm > > > > -- > Marcelo > > > This e-mail (including any attachments) is private and confidential, may > contain proprietary or privileged information and is intended for the named > recipient(s) only. Unintended recipients are strictly prohibited from taking > action on the basis of information in this e-mail and must contact the sender > immediately, delete this e-mail (and all attachments) and destroy any hard > copies. Nomura will not accept responsibility or liability for the accuracy > or completeness of, or the presence of any virus or disabling code in, this > e-mail. If verification is sought please request a hard copy. Any reference > to the terms of executed transactions should be treated as preliminary only > and subject to formal written confirmation by Nomura. Nomura reserves the > right to retain, monitor and intercept e-mail communications through its > networks (subject to and in accordance with applicable laws). No > confidentiality or privilege is waived or lost by Nomura by any > mistransmission of this e-mail. Any reference to "Nomura" is a reference to > any entity in the Nomura Holdings, Inc. group. Please read our Electronic > Communications Legal Notice which forms part of this e-mail: > http://www.Nomura.com/email_disclaimer.htm > -- Marcelo --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
