Matthias Pohl created FLINK-30793:
-------------------------------------
Summary: PyFlink YARN per-job on Docker test fails on Azure due to
permission issues
Key: FLINK-30793
URL: https://issues.apache.org/jira/browse/FLINK-30793
Project: Flink
Issue Type: Bug
Components: Connectors / Hadoop Compatibility, Deployment / YARN
Affects Versions: 1.17.0
Reporter: Matthias Pohl
The following build failed due to some hdfs/yarn permission issues in PyFlink
YARN per-job on Docker e2e test:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45202&view=logs&j=af184cdd-c6d8-5084-0b69-7e9c67b35f7a&t=160c9ae5-96fd-516e-1c91-deb81f59292a&l=10587
{code}
[...]
Jan 26 02:17:31 23/01/26 02:12:20 FATAL hs.JobHistoryServer: Error starting
JobHistoryServer
Jan 26 02:17:31 org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Error
creating done directory:
[hdfs://master.docker-hadoop-cluster-network:9000/tmp/hadoop-yarn/staging/history/done]
Jan 26 02:17:31 at
org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.tryCreatingHistoryDirs(HistoryFileManager.java:698)
Jan 26 02:17:31 at
org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.createHistoryDirs(HistoryFileManager.java:634)
Jan 26 02:17:31 at
org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.serviceInit(HistoryFileManager.java:595)
Jan 26 02:17:31 at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
Jan 26 02:17:31 at
org.apache.hadoop.mapreduce.v2.hs.JobHistory.serviceInit(JobHistory.java:96)
Jan 26 02:17:31 at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
Jan 26 02:17:31 at
org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
Jan 26 02:17:31 at
org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.serviceInit(JobHistoryServer.java:152)
Jan 26 02:17:31 at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
Jan 26 02:17:31 at
org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.launchJobHistoryServer(JobHistoryServer.java:228)
Jan 26 02:17:31 at
org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.main(JobHistoryServer.java:238)
Jan 26 02:17:31 Caused by: org.apache.hadoop.security.AccessControlException:
Permission denied: user=mapred, access=WRITE, inode="/":hdfs:hadoop:drwxr-xr-x
Jan 26 02:17:31 at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:350)
Jan 26 02:17:31 at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:251)
Jan 26 02:17:31 at
org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:189)
Jan 26 02:17:31 at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1756)
Jan 26 02:17:31 at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1740)
Jan 26 02:17:31 at
org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1699)
Jan 26 02:17:31 at
org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:60)
Jan 26 02:17:31 at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3007)
Jan 26 02:17:31 at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1141)
Jan 26 02:17:31 at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:659)
Jan 26 02:17:31 at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
[...]
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
