| Hello, I have been trying to Use StreamingFileSink to write to parquetFiles into azure blob storage. I am getting the following error. I did see in the ticket https://issues.apache.org/jira/browse/FLINK-17444 that support for StreamingFileSink is not yet provided. |
code.java
Description: Binary data
WARNING: An illegal reflective access operation has occurred WARNING: Illegal reflective access by org.apache.flink.api.java.ClosureCleaner (file:/Users/user/.m2/repository/org/apache/flink/flink-core/1.13.0/flink-core-1.13.0.jar) to field java.util.Properties.serialVersionUID WARNING: Please consider reporting this to the maintainers of org.apache.flink.api.java.ClosureCleaner WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations WARNING: All illegal access operations will be denied in a future release 11:43:23,780 INFO org.apache.flink.fs.azurefs.AzureFSFactory [] - Trying to load and instantiate Azure File System 11:43:23,907 INFO org.apache.flink.fs.azure.shaded.org.eclipse.jetty.util.log [] - Logging initialized @6378ms log4j:WARN No appenders could be found for logger (org.apache.flink.fs.shaded.hadoop3.org.apache.commons.beanutils.converters.BooleanConverter). log4j:WARN Please initialize the log4j system properly. log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. 11:43:24,034 WARN org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.metrics2.impl.MetricsConfig [] - Cannot locate configuration: tried hadoop-metrics2-azure-file-system.properties,hadoop-metrics2.properties 11:43:24,076 INFO org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.metrics2.impl.MetricsSystemImpl [] - Scheduled Metric snapshot period at 10 second(s). 11:43:24,076 INFO org.apache.flink.fs.shaded.hadoop3.org.apache.hadoop.metrics2.impl.MetricsSystemImpl [] - azure-file-system metrics system started Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/flink/fs/azure/common/hadoop/HadoopRecoverableWriter at org.apache.flink.fs.azure.common.hadoop.HadoopFileSystem.createRecoverableWriter(HadoopFileSystem.java:215) at org.apache.flink.connector.file.sink.FileSink$BulkFormatBuilder.createBucketWriter(FileSink.java:466) at org.apache.flink.connector.file.sink.FileSink$BulkFormatBuilder.getWriterStateSerializer(FileSink.java:447) at org.apache.flink.connector.file.sink.FileSink.getWriterStateSerializer(FileSink.java:128) at org.apache.flink.streaming.runtime.translators.SinkTransformationTranslator.addWriter(SinkTransformationTranslator.java:178) at org.apache.flink.streaming.runtime.translators.SinkTransformationTranslator.internalTranslate(SinkTransformationTranslator.java:146) at org.apache.flink.streaming.runtime.translators.SinkTransformationTranslator.translateForStreaming(SinkTransformationTranslator.java:103) at org.apache.flink.streaming.runtime.translators.SinkTransformationTranslator.translateForStreaming(SinkTransformationTranslator.java:58) at org.apache.flink.streaming.api.graph.StreamGraphGenerator.translate(StreamGraphGenerator.java:714) at org.apache.flink.streaming.api.graph.StreamGraphGenerator.transform(StreamGraphGenerator.java:446) at org.apache.flink.streaming.api.graph.StreamGraphGenerator.generate(StreamGraphGenerator.java:301) at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getStreamGraph(StreamExecutionEnvironment.java:2010) at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.getStreamGraph(StreamExecutionEnvironment.java:1995) at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1834) at com.grpId.StreamingJob.main(StreamingJob.java:113) Caused by: java.lang.ClassNotFoundException: org.apache.flink.fs.azure.common.hadoop.HadoopRecoverableWriter at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:606) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:168) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522) ... 15 more
Process finished with exit code 1
I am trying to run flink in IntelliJ and created the project using the maven archetype as mentioned in the docs and added flink-azure-fs-hadoop Dependency |
pom.xml
Description: XML document
Kindly let me know what I am missing also Is there a boiler plate code I can refer to? Thank you. Regards, Sudhanva |
