Hey Ken, I have flink-s3-fs-hadoop as a provided dependency in my project, and I've configured my IDE to include provided dependencies when starting applications. Works just fine.
On Tue, Jan 3, 2023 at 5:06 PM Ken Krugler <kkrugler_li...@transpac.com> wrote: > Hi all, > > With Flink 1.15.x, is there a way to use the S3 Presto plugin when running > code on the MiniCluster? > > I can’t just add that jar as a dependency when testing, as I get: > > java.lang.NoClassDefFoundError: Could not initialize class > com.facebook.presto.hive.s3.PrestoS3FileSystem > at > org.apache.flink.fs.s3presto.S3FileSystemFactory.createHadoopFileSystem(S3FileSystemFactory.java:88) > ~[flink-s3-fs-presto-1.15.1.jar:1.15.1] > at > org.apache.flink.fs.s3.common.AbstractS3FileSystemFactory.create(AbstractS3FileSystemFactory.java:126) > ~[flink-s3-fs-presto-1.15.1.jar:1.15.1] > > I assume that’s because of this warning in the Flink docs: > > The s3 file systems (flink-s3-fs-presto and flink-s3-fs-hadoop) can only > be used as plugins as we already removed the relocations. Placing them in > libs/ will result in system failures. > > > In the latest Flink JavaDocs, there’s a way to specify the PluginManager > for the MiniClusterConfiguration, but I don’t see that in Flink 1.15.x > > So is there a workaround to allow me to run a test from inside of my IDE, > using the MiniCluster, that reads from S3? > > Thanks, > > — Ken > > -------------------------- > Ken Krugler > http://www.scaleunlimited.com > Custom big data solutions > Flink, Pinot, Solr, Elasticsearch > >
