Nevermind my last e-mail. HDFS complains about not understanding "3777"...
On Thu, Jan 8, 2015 at 9:46 AM, Marcelo Vanzin <[email protected]> wrote: > Hmm. Can you set the permissions of "/apps/spark/historyserver/logs" > to 3777? I'm not sure HDFS respects the group id bit, but it's worth a > try. (BTW that would only affect newly created log directories.) > > On Thu, Jan 8, 2015 at 1:22 AM, <[email protected]> wrote: >> Hi Vanzin, >> >> I am using the MapR distribution of Hadoop. The history server logs are >> created by a job with the permissions: >> >> drwxrwx--- - <myusername> <mygroup> 2 2015-01-08 09:14 >> /apps/spark/historyserver/logs/spark-1420708455212 >> >> However, the permissions of the higher directories are mapr:mapr and the >> user that runs Spark in our case is a unix ID called mapr (in the mapr >> group). Therefore, this can't read my job event logs as shown above. >> >> >> Thanks, >> Michael >> >> >> -----Original Message----- >> From: Marcelo Vanzin [mailto:[email protected]] >> Sent: 07 January 2015 18:10 >> To: England, Michael (IT/UK) >> Cc: [email protected] >> Subject: Re: Spark History Server can't read event logs >> >> The Spark code generates the log directory with "770" permissions. On top of >> that you need to make sure of two things: >> >> - all directories up to /apps/spark/historyserver/logs/ are readable by the >> user running the history server >> - the user running the history server belongs to the group that owns >> /apps/spark/historyserver/logs/ >> >> I think the code could be more explicitly about setting the group of the >> generated log directories and files, but if you follow the two rules above >> things should work. Also, I recommend setting >> /apps/spark/historyserver/logs/ itself to "1777" so that any user can >> generate logs, but only the owner (or a superuser) can delete them. >> >> >> >> On Wed, Jan 7, 2015 at 7:45 AM, <[email protected]> wrote: >>> Hi, >>> >>> >>> >>> When I run jobs and save the event logs, they are saved with the >>> permissions of the unix user and group that ran the spark job. The >>> history server is run as a service account and therefore can’t read the >>> files: >>> >>> >>> >>> Extract from the History server logs: >>> >>> >>> >>> 2015-01-07 15:37:24,3021 ERROR Client >>> fs/client/fileclient/cc/client.cc:1009 >>> Thread: 1183 User does not have access to open file >>> /apps/spark/historyserver/logs/spark-1420644521194 >>> >>> 15/01/07 15:37:24 ERROR ReplayListenerBus: Exception in parsing Spark >>> event log >>> /apps/spark/historyserver/logs/spark-1420644521194/EVENT_LOG_1 >>> >>> org.apache.hadoop.security.AccessControlException: Open failed for file: >>> /apps/spark/historyserver/logs/spark-1420644521194/EVENT_LOG_1, error: >>> Permission denied (13) >>> >>> >>> >>> Is there a setting which I can change that allows the files to be >>> world readable or at least by the account running the history server? >>> Currently, the job appears in the History Sever UI but only states ‘<Not >>> Started>’. >>> >>> >>> >>> Thanks, >>> >>> Michael >>> >>> >>> This e-mail (including any attachments) is private and confidential, >>> may contain proprietary or privileged information and is intended for >>> the named >>> recipient(s) only. Unintended recipients are strictly prohibited from >>> taking action on the basis of information in this e-mail and must >>> contact the sender immediately, delete this e-mail (and all >>> attachments) and destroy any hard copies. Nomura will not accept >>> responsibility or liability for the accuracy or completeness of, or >>> the presence of any virus or disabling code in, this e-mail. If >>> verification is sought please request a hard copy. Any reference to >>> the terms of executed transactions should be treated as preliminary only >>> and subject to formal written confirmation by Nomura. >>> Nomura reserves the right to retain, monitor and intercept e-mail >>> communications through its networks (subject to and in accordance with >>> applicable laws). No confidentiality or privilege is waived or lost by >>> Nomura by any mistransmission of this e-mail. Any reference to >>> "Nomura" is a reference to any entity in the Nomura Holdings, Inc. >>> group. Please read our Electronic Communications Legal Notice which forms >>> part of this e-mail: >>> http://www.Nomura.com/email_disclaimer.htm >> >> >> >> -- >> Marcelo >> >> >> This e-mail (including any attachments) is private and confidential, may >> contain proprietary or privileged information and is intended for the named >> recipient(s) only. Unintended recipients are strictly prohibited from taking >> action on the basis of information in this e-mail and must contact the >> sender immediately, delete this e-mail (and all attachments) and destroy any >> hard copies. Nomura will not accept responsibility or liability for the >> accuracy or completeness of, or the presence of any virus or disabling code >> in, this e-mail. If verification is sought please request a hard copy. Any >> reference to the terms of executed transactions should be treated as >> preliminary only and subject to formal written confirmation by Nomura. >> Nomura reserves the right to retain, monitor and intercept e-mail >> communications through its networks (subject to and in accordance with >> applicable laws). No confidentiality or privilege is waived or lost by >> Nomura by any mistransmission of this e-mail. Any reference to "Nomura" is a >> reference to any entity in the Nomura Holdings, Inc. group. Please read our >> Electronic Communications Legal Notice which forms part of this e-mail: >> http://www.Nomura.com/email_disclaimer.htm >> > > > > -- > Marcelo -- Marcelo --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
