Steve,
When ResourceManager is submitted with an application, AMLauncher
creates the token YARN_AM_RM_TOKEN (token used between RM and AM). When
ApplicationMaster
is launched, it tries to contact RM for registering request, allocate
request to receive containers, finish request. In all the requests,
ResourceManager does the
authorizeRequest, where it checks if the Current User has the token
YARN_AM_RM_TOKEN, if not throws the *"No AMRMToken". *
ResourceManager for every
yarn.resourcemanager.am-rm-tokens.master-key-rolling-interval-sec rolls the
master key, before rolling it, it has a period
of 1.5 * yarn.am.liveness-monitor.expiry-interval-ms during which if AM
contacts RM with allocate request, RM checks if the AM has the
YARN_AM_RM_TOKEN
prepared using the previous master key, if so, it updates the AM user with
YARN_AM_RM_TOKEN prepared using new master key.
If AM contacts with an YARN_AM_RM_TOKEN which is neither constructed
using current master key nor previous master key, then *"Invalid AMRMToken"*
message is thrown. This
error is the one will happen if AM has not been updated with new RM master
key. [YARN-3103 and YARN-2212 ]
Need your help to find scenario where "No AMRMToken" will happen, an user
added with a token but later that token is missing. Is token removed since
expired?
Thanks,
Prabhu Joseph
On Wed, Feb 10, 2016 at 12:59 AM, Hari Shreedharan <
hshreedha...@cloudera.com> wrote:
> The credentials file approach (using keytab for spark apps) will only
> update HDFS tokens. YARN's AMRM tokens should be taken care of by YARN
> internally.
>
> Steve - correct me if I am wrong here: If the AMRM tokens are disappearing
> it might be a YARN bug (does the AMRM token have a 7 day limit as well? I
> thought that was only for HDFS).
>
>
> Thanks,
> Hari
>
> On Tue, Feb 9, 2016 at 8:44 AM, Steve Loughran <ste...@hortonworks.com>
> wrote:
>
>>
>> On 9 Feb 2016, at 11:26, Steve Loughran <ste...@hortonworks.com> wrote:
>>
>>
>> On 9 Feb 2016, at 05:55, Prabhu Joseph <prabhujose.ga...@gmail.com>
>> wrote:
>>
>> + Spark-Dev
>>
>> On Tue, Feb 9, 2016 at 10:04 AM, Prabhu Joseph <
>> prabhujose.ga...@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> A long running Spark job on YARN throws below exception after
>>> running for few days.
>>>
>>> yarn.ApplicationMaster: Reporter thread fails 1 time(s) in a row.
>>> org.apache.hadoop.yarn.exceptions.YarnException: *No AMRMToken found* for
>>> user prabhu at org.apache.hadoop.yarn.ipc.RPC
>>> Util.getRemoteException(RPCUtil.java:45)
>>>
>>> Do any of the below renew the AMRMToken and solve the issue
>>>
>>> 1. yarn-resourcemanager.delegation.token.max-lifetime increase from 7
>>> days
>>>
>>> 2. Configuring Proxy user:
>>>
>>> <property> <name>hadoop.proxyuser.yarn.hosts</name> <value>*</value>
>>> </property>
>>> <property> <name>hadoop.proxyuser.yarn.groups</name> <value>*</value>
>>> </property>
>>>
>>
>> wouldnt do that: security issues
>>
>>
>>> 3. Can Spark-1.4.0 handle with fix
>>> https://issues.apache.org/jira/browse/SPARK-5342
>>>
>>> spark.yarn.credentials.file
>>>
>>>
>>>
>> I'll say "maybe" there
>>
>>
>> uprated to a no, having looked at the code more
>>
>>
>> How to renew the AMRMToken for a long running job on YARN?
>>>
>>>
>>>
>>
>> AMRM token renewal should be automatic in AM; Yarn sends a message to the
>> AM (actually an allocate() response with no containers but a new token at
>> the tail of the message.
>>
>> i don't see any logging in the Hadoopp code there (AMRMClientImpl); filed
>> YARN-4682 to add a log statement
>>
>> if someone other than me were to supply a patch to that JIRA to add a log
>> statement *by the end of the day* I'll review it and get it in to Hadoop 2.8
>>
>>
>> like I said: I'll get this in to hadoop-2.8 if someone is timely with the
>> diff
>>
>>
>
