Hi Oxlade I think using Flink SQL can conveniently fulfill your requirements.
For S3 Parquet files, you can create a temporary table using a filesystem connector[1] . For Iceberg tables, FlinkSQL can easily integrate with the Iceberg catalog[2]. Therefore, you can use Flink SQL to export S3 files to Iceberg. If you only need field mapping or transformation, I believe using Flink SQL + UDF (User-Defined Functions) would be sufficient to meet your needs. [1]. https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/filesystem/#directory-watching [2]. https://iceberg.apache.org/docs/latest/flink-connector/#table-managed-in-hadoop-catalog Best, Feng On Thu, Nov 23, 2023 at 11:23 PM Oxlade, Dan <dan.oxl...@troweprice.com> wrote: > Hi all, > > > > I’m attempting to create a POC in flink to create a pipeline to stream > parquet to a data warehouse in iceberg format. > > > > Ideally – I’d like to watch a directory in s3 (minio locally) and stream > those to iceberg, doing the appropriate schema mapping/translation. > > > > I guess first; does this sound like a crazy idea? > > Assuming not is anyone able to share examples that might get me going. > I’ve found lots of iceberg and flink sql examples but I think I’ll need > something in java to do the schema mapping. Also some examples reading > parquet for s3 seem a little hard to come by. > > > > I’m aware I’ll need a catalog, I can use nessie for the prototype. I’m > also trying to use minio to get this all working locally but this might > just be adding complexity at the moment. > > > > TIA > > Dan > > T. Rowe Price International Ltd (registered number 3957748) is registered > in England and Wales with its registered office at Warwick Court, 5 > Paternoster Square, London EC4M 7DX. T. Rowe Price International Ltd is > authorised and regulated by the Financial Conduct Authority. The company > has a branch in Dubai International Financial Centre (regulated by the DFSA > as a Representative Office). > > T. Rowe Price (including T. Rowe Price International Ltd and its > affiliates) and its associates do not provide legal or tax advice. Any > tax-related discussion contained in this e-mail, including any attachments, > is not intended or written to be used, and cannot be used, for the purpose > of (i) avoiding any tax penalties or (ii) promoting, marketing, or > recommending to any other party any transaction or matter addressed herein. > Please consult your independent legal counsel and/or professional tax > advisor regarding any legal or tax issues raised in this e-mail. > > The contents of this e-mail and any attachments are intended solely for > the use of the named addressee(s) and may contain confidential and/or > privileged information. Any unauthorized use, copying, disclosure, or > distribution of the contents of this e-mail is strictly prohibited by the > sender and may be unlawful. If you are not the intended recipient, please > notify the sender immediately and delete this e-mail. >
