Hi Moritz, So our specific use case is that we need two different sets of credentials we need to use, one for reading AWS data from one account and one for writing AWS to a separate account. Minus using the Aws ProviderChain (which is *not* supported by the Javadocs), is there any other solution that currently exists within beam that I can leverage? Or is the only option for me to contribute to the open source and write my own custom implementation?
Thanks, Ramya On Mon, Sep 25, 2023 at 2:18 AM Moritz Mack <mm...@apache.org> wrote: > Hi Ramya, > > unfortunately only a subset of AWS credential providers is supported. > Additionally, programmatic configuration is discouraged as there are > options that are hard to support and there's no decent way to validate. > > Please have a look at the Javadocs to see what is supported: > > https://beam.apache.org/releases/javadoc/2.50.0/org/apache/beam/sdk/io/aws2/options/AwsOptions.html > <https://urldefense.com/v3/__https://beam.apache.org/releases/javadoc/2.50.0/org/apache/beam/sdk/io/aws2/options/AwsOptions.html__;!!FrPt2g6CO4Wadw!LuDG8Jj5UPlTf0Yfe6EU0O91a4IgQ3gbf72NLfblspR5vXxivh7zhcfxUDzZSL_Ep0VhE-Kax27VMkJZCHU$> > > Also, there's an open ticket to support custom implementations. That > shouldn't be too hard in case you are interested in contributing to Beam. > https://github.com/apache/beam/issues/26569 > <https://urldefense.com/v3/__https://github.com/apache/beam/issues/26569__;!!FrPt2g6CO4Wadw!LuDG8Jj5UPlTf0Yfe6EU0O91a4IgQ3gbf72NLfblspR5vXxivh7zhcfxUDzZSL_Ep0VhE-Kax27VC8pmsqU$> > > Kind regards, > Moritz > > > Am So., 24. Sept. 2023 um 06:29 Uhr schrieb deepak kumar <kdq...@gmail.com > >: > >> It seems there is problem with the fat/Uber jar . It might have multiple >> versions of the classs being overwritten with duplicates . >> Check this link : >> >> >> https://stackoverflow.com/questions/57039951/beam-failed-to-serialize-and-deserialize-property-awscredentialsprovider >> <https://urldefense.com/v3/__https://stackoverflow.com/questions/57039951/beam-failed-to-serialize-and-deserialize-property-awscredentialsprovider__;!!FrPt2g6CO4Wadw!LuDG8Jj5UPlTf0Yfe6EU0O91a4IgQ3gbf72NLfblspR5vXxivh7zhcfxUDzZSL_Ep0VhE-Kax27VSQs0Zn8$> >> >> >> On Sat, 23 Sep 2023 at 9:55 PM, Ramya Prasad via user < >> user@beam.apache.org> wrote: >> >>> Hello, >>> >>> I am a developer trying to use Apache Beam, and I'm running into an >>> issue with trying to provide multiple AWS credentials to the AwsOptions >>> class within the PipelineOptions class. My use case requires multi-account >>> reads, and I've tried using the AwsCredentialsProviderChain class >>> within the options.as >>> <https://urldefense.com/v3/__http://options.as__;!!FrPt2g6CO4Wadw!LuDG8Jj5UPlTf0Yfe6EU0O91a4IgQ3gbf72NLfblspR5vXxivh7zhcfxUDzZSL_Ep0VhE-Kax27VQLt_c3w$>(AwsOptions.class).setAwsCredentialsProvider() >>> method, but I keep getting the error >>> >>> Caused by: com.fasterxml.jackson.databind.JsonMappingException: >>> Unexpected IOException (of type java.io.IOException): Failed to serialize >>> and deserialize property 'awsCredentialsProvider' with value >>> 'AwsCredentialsProviderChain. >>> >>> Is there anything I'm doing wrong? Any help would be appreciated! >>> >>> Thanks and sincerely, >>> Ramya >>> ------------------------------ >>> >>> The information contained in this e-mail is confidential and/or >>> proprietary to Capital One and/or its affiliates and may only be used >>> solely in performance of work or services for Capital One. The information >>> transmitted herewith is intended only for use by the individual or entity >>> to which it is addressed. If the reader of this message is not the intended >>> recipient, you are hereby notified that any review, retransmission, >>> dissemination, distribution, copying or other use of, or taking of any >>> action in reliance upon this information is strictly prohibited. If you >>> have received this communication in error, please contact the sender and >>> delete the material from your computer. >>> >>> >>> >>> >>> ______________________________________________________________________ The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
