YARN RM keep trying to set new AMRMToken for applications during the master key rolling period

[Paul Lam]

Hi,

AFAIK, YARN RM rolls master keys used to generate AMRMToken periodically (default to 1 day). Before the new master key takes effect, there's a window that the AMs can update its AMRMToken, during which RM would generate a new AMRMToken and send it back to the AM when it detects an AMRMToken using the old master key in the AM reports.

I'm using hadoop-2.6.0-cdh5.6.0 and I find that in the RM logs, during the whole window RM keeping trying to create and set new tokens for the applications until the rolling period is ended, maybe 100+ times for a single application. It seems that the new AMRMToken is ignored by the AM. But it's strange that the applications could run for from days to weeks before it fails with "Invalid AMRMToken" error.

I tried to and didn't find logs to confirm that the AMRMToken is updated on the AM or NM side, so I'm suspecting it the token update fails somehow. Could anyone help me with this? Thanks a lot!

Best,
Paul Lam


---------------------------------------------------------------------
To unsubscribe, e-mail: user-***@hadoop.apache.org
For additional commands, e-mail: user-***@hadoop.apache.org