[
https://issues.apache.org/jira/browse/FLINK-29311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nicolas Pocock updated FLINK-29311:
-----------------------------------
Description:
I am attempting to run the StateFun docker image locally in a docker-compose
stack. I have Kinesis running using Localstack and want to use it as a
consumer.
The documentation says to use a custom endpoint you can use the following
config with the _io.statefun.kinesis.v1/ingress_ module:
{code:java}
awsCredentials:
type: custom-endpoint
endpoint: https://localhost:4567
id: us-west-1{code}
The above errors as the custom endpoint property is on the AWS region instead
so it needs to be:
{code:java}
awsRegion:
id: us-east-1
type: custom-endpoint
endpoint: http://localstack:4566{code}
However, I am unable to use HTTP as it complains that:
{noformat}
Caused by: java.lang.IllegalArgumentException: Invalid service endpoint url:
http://localstack:4566; Only custom service endpoints using HTTPS are
supported{noformat}
I tried using HTTPS but it errors with a certificate issue, I think this is an
issue with Localstack.
Some of the docs for the Kinesis connectors show HTTP being used with a custom
endpoint
[https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/datastream/kinesis/#using-custom-kinesis-endpoints]
It would be nice if we can use HTTP when testing locally
I see from this issue [https://github.com/localstack/localstack/issues/893]
there is an option you can pass there is a configuration option
TRUST_ALL_CERTIFICATES that can be provided to the NettyNioAsyncHttpClient that
allows the above to work.
was:
I am attempting to run the StateFun docker image locally in a docker-compose
stack. I have Kinesis running using Localstack and want to use it as a
consumer.
The documentation says to use a custom endpoint you can use the following
config with the _io.statefun.kinesis.v1/ingress_ module:
{code:java}
awsCredentials:
type: custom-endpoint
endpoint: https://localhost:4567
id: us-west-1{code}
The above errors as the custom endpoint property is on the AWS region instead
so it needs to be:
{code:java}
awsRegion:
id: us-east-1
type: custom-endpoint
endpoint: http://localstack:4566{code}
However, I am unable to use HTTP as it complains that:
{noformat}
Caused by: java.lang.IllegalArgumentException: Invalid service endpoint url:
http://localstack:4566; Only custom service endpoints using HTTPS are
supported{noformat}
I tried using HTTPS but it errors with a certificate issue, I think this is an
issue with Localstack.
Some of the docs for the Kinesis connectors show HTTP being used with a custom
endpoint
[https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/datastream/kinesis/#using-custom-kinesis-endpoints]
It would be nice if we can use HTTP when testing locally
> Unable to use http for Kinesis consumer when testing locally
> ------------------------------------------------------------
>
> Key: FLINK-29311
> URL: https://issues.apache.org/jira/browse/FLINK-29311
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis, Stateful Functions
> Affects Versions: 1.15.2
> Environment: Here is my Kinesis module code
>
> kind: io.statefun.kinesis.v1/ingress
> spec:
> id: com.sentiment
> awsRegion:
> id: us-east-1
> type: custom-endpoint
> customEndpoint: [https://localstack:4566|https://localstack:4566/]
> awsCredentials:
> type: basic
> id: us-east-1
> accessKeyId: key
> secretAccessKey: secret
> startupPosition:
> type: latest
> streams:
> - stream: customer-details
> valueType: SentimentEvent
> targets:
> - sentiment
> clientConfigProperties:
> - SocketTimeout: 9999
> - MaxConnections: 15
> And I am using the latest apache/statefun Docker image within docker compose
> Reporter: Nicolas Pocock
> Priority: Major
>
> I am attempting to run the StateFun docker image locally in a docker-compose
> stack. I have Kinesis running using Localstack and want to use it as a
> consumer.
> The documentation says to use a custom endpoint you can use the following
> config with the _io.statefun.kinesis.v1/ingress_ module:
> {code:java}
> awsCredentials:
> type: custom-endpoint
> endpoint: https://localhost:4567
> id: us-west-1{code}
> The above errors as the custom endpoint property is on the AWS region instead
> so it needs to be:
> {code:java}
> awsRegion:
> id: us-east-1
> type: custom-endpoint
> endpoint: http://localstack:4566{code}
> However, I am unable to use HTTP as it complains that:
> {noformat}
> Caused by: java.lang.IllegalArgumentException: Invalid service endpoint url:
> http://localstack:4566; Only custom service endpoints using HTTPS are
> supported{noformat}
> I tried using HTTPS but it errors with a certificate issue, I think this is
> an issue with Localstack.
> Some of the docs for the Kinesis connectors show HTTP being used with a
> custom endpoint
> [https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/connectors/datastream/kinesis/#using-custom-kinesis-endpoints]
> It would be nice if we can use HTTP when testing locally
> I see from this issue [https://github.com/localstack/localstack/issues/893]
> there is an option you can pass there is a configuration option
> TRUST_ALL_CERTIFICATES that can be provided to the NettyNioAsyncHttpClient
> that allows the above to work.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
