talhakhan119 opened a new issue, #11446:
URL: https://github.com/apache/pinot/issues/11446
I am unable to start my pinot controller pod in AWS EKS environment.
**For that I have added configurations in values.template.yaml for pinot to
connect it from aws s3**
As I have confirmed my serviceaccount have correct permissions and policies
to access AWS S3
As I have tested it while exec into another pod using the same service
account:
```
in the pinot pod
apt-get update
apt-get install awscli
# aws s3 ls s3://perceptdata-pinot
PRE controller-data/
PRE pinot-data/
```
**Configurations:**
```
# Extra configs will be appended to pinot-controller.conf file
extra:
configs: |-
pinot.set.instance.id.to.hostname=true
controller.task.scheduler.enabled=true
controller.disable.ingestion.groovy=false
pinot.controller.storage.factory.class.s3=org.apache.pinot.plugin.filesystem.S3PinotFS
pinot.controller.storage.factory.s3.region=eu-west-1
pinot.controller.segment.fetcher.protocols=file,http,s3
pinot.controller.segment.fetcher.s3.class=org.apache.pinot.common.utils.fetcher.PinotFSSegmentFetcher
```
**The data directory for aws s3 bucket is:**
```
data:
dir: s3://perceptdata-pinot/controller-data
```
The following Error logs I am facing:
```
Initializing ControllerFilePathProvider
Data directory: s3://perceptdata-pinot/controller-data/
Failed to start a Pinot [CONTROLLER] at 17.271 since launch
java.lang.RuntimeException: Caught exception while initializing
ControllerFilePathProvider
at
org.apache.pinot.controller.BaseControllerStarter.initControllerFilePathProvider(BaseControllerStarter.java:569)
~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
org.apache.pinot.controller.BaseControllerStarter.setUpPinotController(BaseControllerStarter.java:380)
~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
org.apache.pinot.controller.BaseControllerStarter.start(BaseControllerStarter.java:328)
~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
org.apache.pinot.tools.service.PinotServiceManager.startController(PinotServiceManager.java:118)
~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
org.apache.pinot.tools.service.PinotServiceManager.startRole(PinotServiceManager.java:87)
~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
org.apache.pinot.tools.admin.command.StartServiceManagerCommand.lambda$startBootstrapServices$0(StartServiceManagerCommand.java:251)
~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startPinotService(StartServiceManagerCommand.java:304)
[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
org.apache.pinot.tools.admin.command.StartServiceManagerCommand.startBootstrapServices(StartServiceManagerCommand.java:250)
[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
org.apache.pinot.tools.admin.command.StartServiceManagerCommand.execute(StartServiceManagerCommand.java:196)
[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
org.apache.pinot.tools.admin.command.StartControllerCommand.execute(StartControllerCommand.java:187)
[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at org.apache.pinot.tools.Command.call(Command.java:33)
[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at org.apache.pinot.tools.Command.call(Command.java:29)
[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at picocli.CommandLine.access$1300(CommandLine.java:145)
[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at picocli.CommandLine.execute(CommandLine.java:2078)
[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
org.apache.pinot.tools.admin.PinotAdministrator.execute(PinotAdministrator.java:171)
[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
org.apache.pinot.tools.admin.PinotAdministrator.main(PinotAdministrator.java:202)
[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
Caused by:
org.apache.pinot.controller.api.resources.InvalidControllerConfigException:
Caught exception while initializing file upload path provider
at
org.apache.pinot.controller.api.resources.ControllerFilePathProvider.<init>(ControllerFilePathProvider.java:107)
~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
org.apache.pinot.controller.api.resources.ControllerFilePathProvider.init(ControllerFilePathProvider.java:49)
~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
org.apache.pinot.controller.BaseControllerStarter.initControllerFilePathProvider(BaseControllerStarter.java:567)
~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
... 20 more
Caused by: software.amazon.awssdk.services.s3.model.S3Exception: Access
Denied (Service: S3, Status Code: 403, Request ID: TMMVB33627KABQ7N, Extended
Request ID:
VOWG6S7PM21PAQ6FNeGPIMFeMFiz/CnWf+MoJ7pyGTktnYuOEV7mzjNGnXe8DXx4ZP1jCDqXnco=)
at
software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleErrorResponse(AwsXmlPredicatedResponseHandler.java:156)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handleResponse(AwsXmlPredicatedResponseHandler.java:106)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:84)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.protocols.xml.internal.unmarshall.AwsXmlPredicatedResponseHandler.handle(AwsXmlPredicatedResponseHandler.java:42)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler$Crc32ValidationResponseHandler.handle(AwsSyncClientHandler.java:94)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.core.internal.handler.BaseClientHandler.lambda$successTransformationResponseHandler$5(BaseClientHandler.java:229)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:128)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:154)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:107)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:162)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:91)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
software.amazon.awssdk.services.s3.DefaultS3Client.listObjectsV2(DefaultS3Client.java:5614)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
org.apache.pinot.plugin.filesystem.S3PinotFS.isDirectory(S3PinotFS.java:572)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
org.apache.pinot.plugin.filesystem.S3PinotFS.exists(S3PinotFS.java:437)
~[pinot-s3-0.12.1-shaded.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
org.apache.pinot.spi.filesystem.NoClosePinotFS.exists(NoClosePinotFS.java:74)
~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
org.apache.pinot.controller.api.resources.ControllerFilePathProvider.<init>(ControllerFilePathProvider.java:71)
~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
org.apache.pinot.controller.api.resources.ControllerFilePathProvider.init(ControllerFilePathProvider.java:49)
~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
at
org.apache.pinot.controller.BaseControllerStarter.initControllerFilePathProvider(BaseControllerStarter.java:567)
~[pinot-all-0.12.1-jar-with-dependencies.jar:0.12.1-6e235a4ec2a16006337da04e118a435b5bb8f6d8]
... 20 more
Shutting down Pinot Service Manager with all running Pinot instances...
Shutting down Pinot Service Manager admin application...
Deregistering service status handler
```
**As I am using a service account to access aws in my pinot-controller pod**
as the specs of service account are as:
```
apiVersion: v1
kind: ServiceAccount
metadata:
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::333870179331:role/pinot
meta.helm.sh/release-name: pinot
meta.helm.sh/release-namespace: pinot-new
creationTimestamp: "2023-08-25T06:54:37Z"
labels:
app: pinot
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/version: 0.2.7
helm.sh/chart: pinot-0.2.7
heritage: Helm
release: pinot
name: pinot
namespace: pinot-new
resourceVersion: "389804078"
uid: 488a7bf2-15fe-4aac-a3ad-e82c5515b3c9
```
This is the policy for pinot :
```
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "PinotAccessPolicy",
"Effect": "Allow",
"Action": [
"*"
],
"Resource": [
"arn:aws:s3:::perceptdata-pinot/*",
"arn:aws:s3:::perceptdata-pinot"
]
}
]
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
