Hi Andreas, I'd check where the exception occurs (not clear from what you posted) and double-check that the part of the system can access the given path deploy-keys/rest.keystore.
The brute-force solution is to manually copy the files onto all worker nodes on the respective directory + potentially the client. On Mon, Apr 19, 2021 at 4:45 PM Hailu, Andreas [Engineering] < andreas.ha...@gs.com> wrote: > Hi Flink team, > > > > I’m trying to configure a Flink on YARN with SSL enabled. I’ve followed > the documentation’s instruction [1] to generate a Keystore and Truststore > locally, and added a the properties to my flink-conf.yaml. > > security.ssl.rest.keystore: /home/user/ssl/deploy-keys/rest.keystore > > security.ssl.rest.truststore: /home/user/ssl/deploy-keys/rest.truststore > > > > I’ve also added the yarnship option so that the keystore and truststore > are deployed as suggested in [1]. > > > > -m yarn-cluster --class <class> [...] -yt /home/user/ssl/deploy-keys/ > > > > However, starting the Flink cluster results in a NoSuchFileException, > > Caused by: java.nio.file.NoSuchFileException: > /home/user/ssl/deploy-keys/rest.keystore > > at > sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) > > at > sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > > at > sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > > at > sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) > > at java.nio.file.Files.newByteChannel(Files.java:361) > > at java.nio.file.Files.newByteChannel(Files.java:407) > > at > java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384) > > at java.nio.file.Files.newInputStream(Files.java:152) > > at > org.apache.flink.runtime.net.SSLUtils.getKeyManagerFactory(SSLUtils.java:266) > > at > org.apache.flink.runtime.net.SSLUtils.createRestNettySSLContext(SSLUtils.java:392) > > at > org.apache.flink.runtime.net.SSLUtils.createRestNettySSLContext(SSLUtils.java:365) > > at > org.apache.flink.runtime.net.SSLUtils.createRestServerSSLEngineFactory(SSLUtils.java:163) > > at > org.apache.flink.runtime.rest.RestServerEndpointConfiguration.fromConfiguration(RestServerEndpointConfiguration.java:160) > > > > I’m able to see in launch_container.sh that the shipped directory was able > to be created successfully: > > > > mkdir -p deploy-keys > > ln -sf > "/fs/htmp/yarn/local/usercache/delp/appcache/application_1618711298408_2664/filecache/16/rest.truststore" > "deploy-keys/rest.truststore" > > mkdir -p deploy-keys > > ln -sf > "/fs/htmp/yarn/local/usercache/delp/appcache/application_1618711298408_2664/filecache/13/rest.keystore" > "deploy-keys/rest.keystore" > > > > So given the above logs, I tried editing flink-conf.yaml to reflect what I > saw: > > security.ssl.rest.keystore: deploy-keys/rest.keystore > > security.ssl.rest.truststore: deploy-keys/rest.truststore > > > > But that didn’t seem to work, either: > > Caused by: java.nio.file.NoSuchFileException: deploy-keys/rest.truststore > > at > sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) > > at > sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > > at > sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > > at > sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214) > > at java.nio.file.Files.newByteChannel(Files.java:361) > > at java.nio.file.Files.newByteChannel(Files.java:407) > > at > java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384) > > at java.nio.file.Files.newInputStream(Files.java:152) > > at > org.apache.flink.runtime.net.SSLUtils.getTrustManagerFactory(SSLUtils.java:233) > > at > org.apache.flink.runtime.net.SSLUtils.createRestNettySSLContext(SSLUtils.java:397) > > at > org.apache.flink.runtime.net.SSLUtils.createRestNettySSLContext(SSLUtils.java:365) > > at > org.apache.flink.runtime.net.SSLUtils.createRestClientSSLEngineFactory(SSLUtils.java:181) > > at > org.apache.flink.runtime.rest.RestClientConfiguration.fromConfiguration(RestClientConfiguration.java:106) > > > > What needs to be done to get the YARN application to point to the right > keystore and truststore? > > > > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/security-ssl.html#tips-for-yarn--mesos-deployment > > > > ____________ > > > > *Andreas Hailu* > > *Data Lake Engineering *| Goldman Sachs & Co. > > > > ------------------------------ > > Your Personal Data: We may collect and process information about you that > may be subject to data protection laws. For more information about how we > use and disclose your personal data, how we protect your information, our > legal basis to use your information, your rights and who you can contact, > please refer to: www.gs.com/privacy-notices >
