Have a look at the default impl, it's fairly simple: https://github.com/apache/beam/blob/master/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/DefaultS3FileSystemSchemeRegistrar.java
Your registrar has to provide one or two custom configurations using a new uri scheme each, e.g. s3a, s3b for account a and b https://github.com/apache/beam/blob/master/sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/s3/S3FileSystemConfiguration.java#L41 /M Am Di., 26. Sept. 2023 um 17:52 Uhr schrieb Ramya Prasad via user < user@beam.apache.org>: > Gotcha, is there any way you can provide me with a simple example using > the S3FileSystemSchemeRegistrar within the context of a Beam pipeline? > I'm having trouble understanding how to register the schemes, configure the > s3 client builders, and make the pipeline aware of these schemes. > > Thanks! > Ramya > > On Tue, Sep 26, 2023 at 1:40 AM Moritz Mack <mm...@apache.org> wrote: > >> Actually, I doubt a provider chain can solve your problem. it will >> always return the credentials of the first provider in the chain that can >> provide some regardless if these are the correct ones for the usage. >> A typical way to do this is to set up cross-account credentials for AWS. >> >> I assume you are working with S3? As a workaround, if cross-account >> credentials can't be done for you, you could also register separate schemes >> for the two accounts (see S3FileSystemSchemeRegistrar). >> For each scheme you can then configure the S3 client builder accordingly. >> >> / M >> >> Am Mo., 25. Sept. 2023 um 23:23 Uhr schrieb Ramya Prasad via user < >> user@beam.apache.org>: >> >>> Hi Moritz, >>> So our specific use case is that we need two different sets of >>> credentials we need to use, one for reading AWS data from one account and >>> one for writing AWS to a separate account. Minus using the Aws >>> ProviderChain (which is *not* supported by the Javadocs), is there any >>> other solution that currently exists within beam that I can leverage? Or is >>> the only option for me to contribute to the open source and write my own >>> custom implementation? >>> >>> Thanks, >>> Ramya >>> >>> On Mon, Sep 25, 2023 at 2:18 AM Moritz Mack <mm...@apache.org> wrote: >>> >>>> Hi Ramya, >>>> >>>> unfortunately only a subset of AWS credential providers is supported. >>>> Additionally, programmatic configuration is discouraged as there are >>>> options that are hard to support and there's no decent way to validate. >>>> >>>> Please have a look at the Javadocs to see what is supported: >>>> >>>> https://beam.apache.org/releases/javadoc/2.50.0/org/apache/beam/sdk/io/aws2/options/AwsOptions.html >>>> <https://urldefense.com/v3/__https://beam.apache.org/releases/javadoc/2.50.0/org/apache/beam/sdk/io/aws2/options/AwsOptions.html__;!!FrPt2g6CO4Wadw!LuDG8Jj5UPlTf0Yfe6EU0O91a4IgQ3gbf72NLfblspR5vXxivh7zhcfxUDzZSL_Ep0VhE-Kax27VMkJZCHU$> >>>> >>>> Also, there's an open ticket to support custom implementations. That >>>> shouldn't be too hard in case you are interested in contributing to Beam. >>>> https://github.com/apache/beam/issues/26569 >>>> <https://urldefense.com/v3/__https://github.com/apache/beam/issues/26569__;!!FrPt2g6CO4Wadw!LuDG8Jj5UPlTf0Yfe6EU0O91a4IgQ3gbf72NLfblspR5vXxivh7zhcfxUDzZSL_Ep0VhE-Kax27VC8pmsqU$> >>>> >>>> Kind regards, >>>> Moritz >>>> >>>> >>>> Am So., 24. Sept. 2023 um 06:29 Uhr schrieb deepak kumar < >>>> kdq...@gmail.com>: >>>> >>>>> It seems there is problem with the fat/Uber jar . It might have >>>>> multiple versions of the classs being overwritten with duplicates . >>>>> Check this link : >>>>> >>>>> >>>>> https://stackoverflow.com/questions/57039951/beam-failed-to-serialize-and-deserialize-property-awscredentialsprovider >>>>> <https://urldefense.com/v3/__https://stackoverflow.com/questions/57039951/beam-failed-to-serialize-and-deserialize-property-awscredentialsprovider__;!!FrPt2g6CO4Wadw!LuDG8Jj5UPlTf0Yfe6EU0O91a4IgQ3gbf72NLfblspR5vXxivh7zhcfxUDzZSL_Ep0VhE-Kax27VSQs0Zn8$> >>>>> >>>>> >>>>> On Sat, 23 Sep 2023 at 9:55 PM, Ramya Prasad via user < >>>>> user@beam.apache.org> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> I am a developer trying to use Apache Beam, and I'm running into an >>>>>> issue with trying to provide multiple AWS credentials to the AwsOptions >>>>>> class within the PipelineOptions class. My use case requires >>>>>> multi-account >>>>>> reads, and I've tried using the AwsCredentialsProviderChain class >>>>>> within the options.as >>>>>> <https://urldefense.com/v3/__http://options.as__;!!FrPt2g6CO4Wadw!LuDG8Jj5UPlTf0Yfe6EU0O91a4IgQ3gbf72NLfblspR5vXxivh7zhcfxUDzZSL_Ep0VhE-Kax27VQLt_c3w$>(AwsOptions.class).setAwsCredentialsProvider() >>>>>> method, but I keep getting the error >>>>>> >>>>>> Caused by: com.fasterxml.jackson.databind.JsonMappingException: >>>>>> Unexpected IOException (of type java.io.IOException): Failed to serialize >>>>>> and deserialize property 'awsCredentialsProvider' with value >>>>>> 'AwsCredentialsProviderChain. >>>>>> >>>>>> Is there anything I'm doing wrong? Any help would be appreciated! >>>>>> >>>>>> Thanks and sincerely, >>>>>> Ramya >>>>>> ------------------------------ >>>>>> >>>>>> The information contained in this e-mail is confidential and/or >>>>>> proprietary to Capital One and/or its affiliates and may only be used >>>>>> solely in performance of work or services for Capital One. The >>>>>> information >>>>>> transmitted herewith is intended only for use by the individual or entity >>>>>> to which it is addressed. If the reader of this message is not the >>>>>> intended >>>>>> recipient, you are hereby notified that any review, retransmission, >>>>>> dissemination, distribution, copying or other use of, or taking of any >>>>>> action in reliance upon this information is strictly prohibited. If you >>>>>> have received this communication in error, please contact the sender and >>>>>> delete the material from your computer. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> ------------------------------ >>> >>> The information contained in this e-mail is confidential and/or >>> proprietary to Capital One and/or its affiliates and may only be used >>> solely in performance of work or services for Capital One. The information >>> transmitted herewith is intended only for use by the individual or entity >>> to which it is addressed. If the reader of this message is not the intended >>> recipient, you are hereby notified that any review, retransmission, >>> dissemination, distribution, copying or other use of, or taking of any >>> action in reliance upon this information is strictly prohibited. If you >>> have received this communication in error, please contact the sender and >>> delete the material from your computer. >>> >>> >>> >>> >>> ------------------------------ > > The information contained in this e-mail is confidential and/or > proprietary to Capital One and/or its affiliates and may only be used > solely in performance of work or services for Capital One. The information > transmitted herewith is intended only for use by the individual or entity > to which it is addressed. If the reader of this message is not the intended > recipient, you are hereby notified that any review, retransmission, > dissemination, distribution, copying or other use of, or taking of any > action in reliance upon this information is strictly prohibited. If you > have received this communication in error, please contact the sender and > delete the material from your computer. > > > > >
