Hello, just an update. I finally got it working. There were two issues:
* Even we're not using HDFS, we need the core-site.xml, where we define the storage layer (minio in our case). * Apart from that, before the PutHiveQL, the queue still contained some elements pointing to an "adl://..." instead "s3a://...", so each time PutHiveQL picked one, it failed. After referencing the core-site.xml and empty the queues, everything works. Best, ________________________________ De: FABIAN Juan-antonio <juan-antonio.fab...@alstomgroup.com> Enviado: viernes, 24 de enero de 2020 11:24 Para: user@hive.apache.org <user@hive.apache.org> Asunto: Hive expection: Class org.apache.hadoop.fs.adl.AdlFileSystem not found Hello, I'm getting an exception when running a NiFi flow, using the PutHiveQL processor: 10:12:58 UTC ERROR 5f9d00fe-e6dd-3c05-8698-50cab74ee142 nifi-node-0:8080 PutHiveQL[id=5f9d00fe-e6dd-3c05-8698-50cab74ee142] Failed to process session due to Failed to process StandardFlowFileRecord[uuid=8bd48767-dbb3-42c1-8492-4063e1aee6ec,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1579790709540-2, container=default, section=2], offset=448724, length=182],offset=0,name=3613831121596506,size=182] due to java.sql.SQLException: org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.adl.AdlFileSystem not found);: org.apache.nifi.processor.exception.ProcessException: Failed to process StandardFlowFileRecord[uuid=8bd48767-dbb3-42c1-8492-4063e1aee6ec,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1579790709540-2, container=default, section=2], offset=448724, length=182],offset=0,name=3613831121596506,size=182] due to java.sql.SQLException: org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: MetaException(message:java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.adl.AdlFileSystem not found); I've posted this question on the NiFi users list, but maybe you can help me a bit (I'm new both to Hive and NiFi). I had a NiFi cluster with a flow using PutHDFS, and after that, I used PutHiveQL to addPartition. Now I've setup another NiFi cluster, with git as the flow provider, and we replaced HDFS with Minio, so instead using PutParquet/PutHDFS, I'm using PutS3Object. AFAIK, PutHiveQL is talking to our Thrift server, so no need for Hadoop or AzureDataLake dependencies. However, I'm getting this. Both NiFi clusters are deployed on a custom Kubernetes installation on Azure. I'm kind of lost here, so any help is much appreciated. Thanks Juan A. Fabián Simón Data Engineer Alstom Calle Martínez Villergas 49, ed. V - 28027 Madrid - Spain Office: +34 91 384 89 00 Email: <mailto:juan-antonio.fab...@alstomgroup.com> juan-antonio.fab...@alstomgroup.com www.alstom.com<http://www.alstom.com> [cid:850c3903-2713-45c3-9040-c2f64d22bc89] [cid:3539e2fe-0e83-445a-be92-2063029e4590] <https://twitter.com/Alstom> [cid:fa891413-1d00-4c29-8789-4fe7149b2bef] <https://www.linkedin.com/company/alstom/> [cid:34c4d7b6-85be-4793-8910-9805a338456b] <https://www.facebook.com/ALSTOM/> [cid:c542d5f3-d10b-4fda-8c03-fdc0d155e987] <https://www.instagram.com/alstom> [cid:f6b0d063-0c70-4343-85ef-57dc6b720b3d] <[The%20URL%20sent%20was%20deleted%20due%20to%20the%20non-respect%20of%20Alstom%20policies]> ________________________________ CONFIDENTIALITY : This e-mail and any attachments are confidential and may be privileged. If you are not a named recipient, please notify the sender immediately and do not disclose the contents to another person, use it for any purpose or store or copy the information in any medium. ________________________________ Some URLs of this mail have been deactivated due to non-respect of Alstom s policies ________________________________ CONFIDENTIALITY : This e-mail and any attachments are confidential and may be privileged. If you are not a named recipient, please notify the sender immediately and do not disclose the contents to another person, use it for any purpose or store or copy the information in any medium.
