Hi,

环境:flink-1.14.0,单节点standalone



https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/filesystems/s3/

参考官方文档,执行下面命令:

mkdir plugins/s3-fs-hadoop

cp opt/flink-s3-fs-hadoop-1.14.0.jar plugins/s3-fs-hadoop/



在flink-conf中配置了hadoop的路径(s3使用了hadoop的配置文件)

env.hadoop.conf.dir: /data/hadoop/s3

然后启动集群,成功启动后,执行./bin/sql-client.sh,用SQL测试读取s3数据, 出现连接超时报错



但是把flink-s3-fs-hadoop-1.14.0.jar挪到flink的lib下,重启集群,重新执行同样的测试,就可以读取到数据了,其余的配置都没有修改
所以感觉这个plugins目录没有生效?这个plugins和lib目录的区别在哪里,应该如何使用??


附上报错信息:
[ERROR] Could not execute SQL statement. Reason:
org.apache.flink.util.SerializedThrowable: connect timed out


Caused by: com.amazonaws.SdkClientException: Failed to connect to service 
endpoint: 
at 
com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100)
at 
com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:70)
at 
com.amazonaws.internal.InstanceMetadataServiceResourceFetcher.readResource(InstanceMetadataServiceResourceFetcher.java:75)
at 
com.amazonaws.internal.EC2ResourceFetcher.readResource(EC2ResourceFetcher.java:66)
at 
com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsEndpoint(InstanceMetadataServiceCredentialsFetcher.java:58)
at 
com.amazonaws.auth.InstanceMetadataServiceCredentialsFetcher.getCredentialsResponse(InstanceMetadataServiceCredentialsFetcher.java:46)
at 
com.amazonaws.auth.BaseCredentialsFetcher.fetchCredentials(BaseCredentialsFetcher.java:112)
at 
com.amazonaws.auth.BaseCredentialsFetcher.getCredentials(BaseCredentialsFetcher.java:68)
at 
com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:165)
at 
org.apache.hadoop.fs.s3a.AWSCredentialProviderList.getCredentials(AWSCredentialProviderList.java:137)
... 49 more
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at 
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:607)
at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
at sun.net.www.http.HttpClient.New(HttpClient.java:339)
at sun.net.www.http.HttpClient.New(HttpClient.java:357)
at 
sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1226)
at 
sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1205)
at 
sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1056)
at 
sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:990)
at 
com.amazonaws.internal.ConnectionUtils.connectToEndpoint(ConnectionUtils.java:52)
at 
com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:80)
... 58 more

回复