Hi Qing, You may have some luck with using per-bucket S3 configuration. Assuming you're using the flink-s3-fs-hadoop plugin, you should be able to apply different access keys to different buckets, eg: https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#Configuring_different_S3_buckets_with_Per-Bucket_Configuration
I'm not certain if the 's3.<bucket>.<property>' syntax will work, you may need to set the 'fs.s3a.<bucket>.<property>' values directly. Josh On Thu, Jan 18, 2024 at 6:02 AM Qing Lim <q....@mwam.com> wrote: > Hi Jun > > > > I am indeed talking about processing two different tables, but I don’t see > any option that allow configuring credentials at Flink table level, do you > know where is it documented? > > > > Today we are setting the credentials via Flink conf yaml, which is > documented here: > https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/filesystems/s3/#configure-access-credentials > > > > Thanks > > > > *From:* Jun Qin <qinjunje...@gmail.com> > *Sent:* 18 January 2024 10:51 > *To:* User <user@flink.apache.org>; Qing Lim <q....@mwam.com> > *Subject:* Re: Use different S3 access key for different S3 bucket > > > > Hi Qing > > The S3 credentials are associated with Flink SQL tables. > > I assume you are talking about processing/joining from two different > tables, backed up by two different S3 buckets. If so, you can provide > different credentials for different tables, then use the two tables in your > pipeline. > > > > Jun > > On 18. Jan 2024 at 11:32 +0100, Qing Lim <q....@mwam.com>, wrote: > > Hi, I am using Flink SQL to create table backed by S3 buckets. > > > > We are not using AWS S3, so we have to use access key and secret for Auth. > > > > My pipeline depends on 2 different buckets, each requires different > credentials, can flink support this? > > > > > > > > *Qing Lim* | Marshall Wace LLP, George House, 131 Sloane Street, London, > SW1X 9AT | E-mail: q....@mwam.com | Tel: +44 207 925 4865 > > > > This e-mail and any attachments are confidential to the addressee(s) and > may contain information that is legally privileged and/or confidential. If > you are not the intended recipient of this e-mail you are hereby notified > that any dissemination, distribution, or copying of its content is strictly > prohibited. If you have received this message in error, please notify the > sender by return e-mail and destroy the message and all copies in your > possession. > > > To find out more details about how we may collect, use and share your > personal information, please see https://www.mwam.com/privacy-policy. > This includes details of how calls you make to us may be recorded in order > for us to comply with our legal and regulatory obligations. > > > To the extent that the contents of this email constitutes a financial > promotion, please note that it is issued only to and/or directed only at > persons who are professional clients or eligible counterparties as defined > in the FCA Rules. Any investment products or services described in this > email are available only to professional clients and eligible > counterparties. Persons who are not professional clients or eligible > counterparties should not rely or act on the contents of this email. > > > Marshall Wace LLP is authorised and regulated by the Financial Conduct > Authority. Marshall Wace LLP is a limited liability partnership registered > in England and Wales with registered number OC302228 and registered office > at George House, 131 Sloane Street, London, SW1X 9AT. If you are receiving > this e-mail as a client, or an investor in an investment vehicle, managed > or advised by Marshall Wace North America L.P., the sender of this e-mail > is communicating with you in the sender's capacity as an associated or > related person of Marshall Wace North America L.P. ("MWNA"), which is > registered with the US Securities and Exchange Commission ("SEC") as an > investment adviser. Registration with the SEC does not imply that MWNA or > its employees possess a certain level of skill or training. > >
