[
https://issues.apache.org/jira/browse/FLINK-23389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17449139#comment-17449139
]
Nicolaus Weidner commented on FLINK-23389:
------------------------------------------
[~hhkkxxx133] I noticed that the two integration tests,
[GlueSchemaRegistryAvroKinesisITCase|https://github.com/apache/flink/blob/master/flink-end-to-end-tests/flink-glue-schema-registry-avro-test/src/test/java/org/apache/flink/glue/schema/registry/test/GlueSchemaRegistryAvroKinesisITCase.java]
and
[GlueSchemaRegistryJsonKinesisITCase|https://github.com/apache/flink/blob/master/flink-end-to-end-tests/flink-glue-schema-registry-json-test/src/test/java/org/apache/flink/glue/schema/registry/test/json/GlueSchemaRegistryJsonKinesisITCase.java],
are not executed on Azure CI runs, because the access key and secret key env
variables are not present, see e.g. [this
run|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=26986&view=logs&j=af184cdd-c6d8-5084-0b69-7e9c67b35f7a&t=160c9ae5-96fd-516e-1c91-deb81f59292a&l=15852].
Then, during recent testing on Github Actions, we noticed that the tests don't
work even if the env variables are present because AWS expects different
variable names (it expects AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY). See
[this
run|https://github.com/ververica/flink/runs/4301833493?check_suite_focus=true#step:13:17885].
{code:java}
Nov 23 18:40:46 Caused by:
software.amazon.awssdk.core.exception.SdkClientException: Unable to load
credentials from any of the providers in the chain
AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(),
EnvironmentVariableCredentialsProvider(),
WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(),
ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) :
[SystemPropertyCredentialsProvider(): Unable to load credentials from system
settings. Access key must be specified either via environment variable
(AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId).,
EnvironmentVariableCredentialsProvider(): Unable to load credentials from
system settings. Access key must be specified either via environment variable
(AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId).,
WebIdentityTokenCredentialsProvider(): Either the environment variable
AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must
be set., ProfileCredentialsProvider(): Profile file contained no credentials
for profile 'default': ProfileFile(profiles=[]),
ContainerCredentialsProvider(): Cannot fetch credentials from container -
neither AWS_CONTAINER_CREDENTIALS_FULL_URI or
AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set.,
InstanceProfileCredentialsProvider(): The requested metadata is not found at
http://169.254.169.254/latest/meta-data/iam/security-credentials/]
{code}
Finally, even with correct env variable naming, the test still fails because of
an assertion error. It looks like only the first record ever makes it to the
results. See [this
run|https://github.com/ververica/flink/runs/4315084463?check_suite_focus=true#step:13:5317]
for the error, also posted here for convenience:
{code:java}
Nov 24 18:00:55 java.lang.AssertionError: Results received from
'gsr_json_output_stream':
[JsonDataWithSchema(schema={"$id":"https://example.com/address.schema.json","$schema":"http://json-schema.org/draft-07/schema#","type":"object","properties":{"f1":{"type":"string"},"f2":{"type":"integer","maximum":10000}}},
payload={"f1":"olympic","f2":2020})] expected:<8> but was:<1>
{code}
Can you help with fixing these tests?
> AWS Glue Schema Registry JSON support for Apache Flink
> ------------------------------------------------------
>
> Key: FLINK-23389
> URL: https://issues.apache.org/jira/browse/FLINK-23389
> Project: Flink
> Issue Type: New Feature
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
> Reporter: Kexin Hui
> Assignee: Kexin Hui
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> AWS Glue Schema Registry client library has recently released a new version
> (v1.1.1) which highlights JSON support. This request is to add the new data
> format support to launch an integration for Apache Flink with the latest AWS
> Glue Schema Registry.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
