[
https://issues.apache.org/jira/browse/NIFI-13192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18032532#comment-18032532
]
Peter Turcsanyi commented on NIFI-13192:
----------------------------------------
PR is available: https://github.com/apache/nifi/pull/10460
----
AWS SDK for Java v2 does not (or not fully) support all features we used in the
S3 processors with the v1 client. This led to some incompatible changes in the
processors:
*1. Custom signers*
The v1 client library supported the AWS provided v2 and v4 request signers and
NIFI-10969 also introduced an extension point for hooking custom
implementations (typically extensions of v4). Signature handling was
reimplemented in the v2 client library: only the v4 signer is supported and it
is not (really) extensible.
I removed the corresponding properties: _'Signer Override'_, _'Custom Signer
Class Name'_, _'Custom Signer Module Location'_.
A follow-up jira can be created to investigate the feasibility of custom
signers in v2 in more detail.
*2. Region handling*
In the previous version of the S3 processors, the region could be specified as:
- _'Region'_ property (containing the AWS regions)
- _'s3.region'_ FlowFile attribute (_'Region'_ property must be set to _'Use
s3.region Attribute'_)
- via _'Endpoint Override URL'_ property parsed by the v1 client library
(_'Region'_ property is omitted in this case)
The v2 library does not parse the endpoint URL for the region. Instead, the
region must be specified explicitly in all cases.
I added _'Use Custom Region'_ option to the _'Region'_ property and the
dependent _'Custom Region'_ property. The user must fill in this property when
migrating an existing flow. I don't think migration code can be added here
because custom Endpoint URLs can have various forms and it is not
straightforward to parse the region from them.
The new _'Use Custom Region'_ option + _'Custom Region'_ property can cover the
existing _'s3.region'_ FlowFile attribute use case as well, so I removed that
option and added automatic migration: _'Use s3.region Attribute' => 'Use Custom
Region' + 'Custom Region' = ${s3.region}_. Also, any FlowFile attribute can be
used from now on.
The _'Endpoint Override URL' + 'Custom Region'_ case is applicable to the other
AWS processors as well, so I added _'Custom Region'_ in all processors.
_'s3.region'_ FlowFile attribute handling is specific to the S3 processors, so
the _'Custom Region'_ property supports Expression Language with FlowFile
attributes only in the S3 processors (can be extended to other processors
later).
*3. KMS region in Client-side KMS encryption:*
The new encryption client does not allow to separately specify the region of
the KMS key. It must be in the same region as the S3 bucket.
I removed the _'KMS Region'_ property from StandardS3EncryptionService.
----
I also applied further property changes. These are not strictly related to v2
but it was a good occasion to do some clean-up. All changes are backward
compatible because migration code was added in the components.
*ACL properties:*
I kept only those properties that have real effect. _'Owner'_ and _'Write
Permission User List'_ have been removed completely because it is not possible
to set the owner and the write user explicitly (using either the v1 or the v2
client library). Other ACL properties are kept only on the Put and Copy
processors because setting the ACLs is applicable only there (and not for List,
Fetch, Get and Delete).
*PutS3Object: Server Side Encryption*
The property has been redundant since the introduction of
AmazonS3EncryptionService. SSE_S3 strategy covers the same use case. If the
property is true, it will be migrated to StandardS3EncryptionService.
*PutS3Object: Expiration Time Rule*
The property had no effect because the expiration rule can only be assigned via
expiration policies on the service side but not explicitly on the client side.
The property has been removed.
*StandardS3EncryptionService: Key ID or Key Material*
The service contained this multi-purpose property from the time before
dependsOn(). It has been separated to _'KMS Key ID'_ and _'Key Material'_
properties, displayed conditionally based on _'Encryption Strategy'_.
> Upgrade AWS S3 Processors to use AWS 2.x libraries
> --------------------------------------------------
>
> Key: NIFI-13192
> URL: https://issues.apache.org/jira/browse/NIFI-13192
> Project: Apache NiFi
> Issue Type: Improvement
> Reporter: Joe Gresock
> Assignee: Peter Turcsanyi
> Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> It looks like AWS recently began supporting client side encryption in the v2
> SDK: https://github.com/aws/aws-sdk-java-v2/issues/34
> This will enable us to upgrade the S3 processors without losing functionality.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
