CandiceSu opened a new issue, #3106:
URL: https://github.com/apache/parquet-java/issues/3106
### Describe the bug, including details regarding any error messages,
version, and platform.
Hi we created one AvroParquetWriter to write to aws s3.
This is how we create the writer:
`ParquetWriter<GenericRecord> writer =
AvroParquetWriter.<GenericRecord>builder(path)
.withConf(conf)
.withWriteMode(ParquetFileWriter.Mode.OVERWRITE)
.withSchema(avroSchema)
.build()`
And this is how we create the conf:
` AwsCredentialsProvider credentialsProvider =
DefaultCredentialsProvider.builder().build();
var credentials = credentialsProvider.resolveCredentials();
AwsSessionCredentials sessionCredentials = (AwsSessionCredentials)
credentials;
Configuration conf = new Configuration();
conf.set("fs.s3a.access.key", sessionCredentials.accessKeyId());
conf.set("fs.s3a.secret.key", sessionCredentials.secretAccessKey());
conf.set("fs.s3a.session.token", sessionCredentials.sessionToken());
conf.set("fs.s3a.endpoint", "s3.my_region.amazonaws.com");
`
And we are running a streaming application in ECS, each time a task come in,
we get a new conf and create a new writer.
We see the token and everything are correct and refreshed each time, but
writer fail to write to s3 and throw
'software.amazon.awssdk.services.s3.model.S3Exception: The provided token has
expired' after the application running for few hours.
So it looks like the writer, once initialized 1st time, it keep using the
1st time configuration - even if we closed the writer, cleared cache, create
new writer with new conf... And after few hours of course you hit token expire
issue when that 1st time token expires.
We even did a small test - create writer 1st time with correct conf, write
to s3 works. Close 1st writer, create writer 2nd time with invalid conf, write
to s3 still works!!!
Is there anything wrong in the way we initialize the writer or conf?
Thanks!
### Component(s)
Avro
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@parquet.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@parquet.apache.org
For additional commands, e-mail: issues-h...@parquet.apache.org
