I’m using a new flume sink to S3 that doesn’t seem to successfully close out
.tmp files created in S3 buckets. So I’m essentially getting a whole lot of
unclosed .tmp files.
The IAM role being used has full S3 permissions to this bucket.
Here’s the flume error when trying to rename and close th
Could you please share the IAM policy attached to the role granting permission
to the bucket, as well the bucket policy, if one is present?
Please remove or obfuscate bucket names, account number, etc.
The policy on the role or bucket is most certainly a missing permission, rename
requires a fe
Here you go. Names changed to protect the innocent. :-)
{
"Version": "2012-10-17",
"Id": "Policy1527067401408",
"Statement": [
{
"Sid": "AccessForEventServerRole",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::X
Tahnks, bucket policy looks good...
Are any denies present on the policies attached to event-server-s3-role??
Are you able to aws s3 mv s3://my-bucket-name/file.tmp
s3://my-bucket-name/file from the instance? Not sure if that's a valid test
for what flume/aws-sdk are doing underneath but might re
Well I think I have this figured out.
I had to change the sink to use “s3n”, instead of “s3a”, and add the AWS key
and secret key to the core-site.xml to make “s3n” work properly. Then with a
change to the bucket policy to allow the IAM user (e.g. keys) full perms to
that bucket. I’m no longer
