could the provided scope be the issue?
On Sun, Aug 27, 2023 at 2:58 PM Dipayan Dev <dev.dipaya...@gmail.com> wrote:
> Using the following dependency for Spark 3 in POM file (My Scala version
> is 2.12.14)
>
>
>
>
>
>
> *<dependency> <groupId>org.elasticsearch</groupId>
> <artifactId>elasticsearch-spark-30_2.12</artifactId>
> <version>7.12.0</version> <scope>provided</scope></dependency>*
>
>
> The code throws error at this line :
> df.write.format("es").mode("overwrite").options(elasticOptions).save("index_name")
> The same code is working with Spark 2.4.0 and the following dependency
>
>
>
>
>
> *<dependency> <groupId>org.elasticsearch</groupId>
> <artifactId>elasticsearch-spark-20_2.12</artifactId>
> <version>7.12.0</version></dependency>*
>
>
> On Mon, 28 Aug 2023 at 12:17 AM, Holden Karau <hol...@pigscanfly.ca>
> wrote:
>
>> What’s the version of the ES connector you are using?
>>
>> On Sat, Aug 26, 2023 at 10:17 AM Dipayan Dev <dev.dipaya...@gmail.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> We're using Spark 2.4.x to write dataframe into the Elasticsearch index.
>>> As we're upgrading to Spark 3.3.0, it throwing out error
>>> Caused by: java.lang.ClassNotFoundException: es.DefaultSource
>>> at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:476)
>>> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:589)
>>> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
>>>
>>> Looking at a few responses from Stackoverflow
>>> <https://stackoverflow.com/a/66452149>. it seems this is not yet
>>> supported by Elasticsearch-hadoop.
>>>
>>> Does anyone have experience with this? Or faced/resolved this issue in
>>> Spark 3?
>>>
>>> Thanks in advance!
>>>
>>> Regards
>>> Dipayan
>>>
>> --
>> Twitter: https://twitter.com/holdenkarau
>> Books (Learning Spark, High Performance Spark, etc.):
>> https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9>
>> YouTube Live Streams: https://www.youtube.com/user/holdenkarau
>>
>
--
CONFIDENTIALITY NOTICE: This electronic communication and any files
transmitted with it are confidential, privileged and intended solely for
the use of the individual or entity to whom they are addressed. If you are
not the intended recipient, you are hereby notified that any disclosure,
copying, distribution (electronic or otherwise) or forwarding of, or the
taking of any action in reliance on the contents of this transmission is
strictly prohibited. Please notify the sender immediately by e-mail if you
have received this email by mistake and delete this email from your system.
Is it necessary to print this email? If you care about the environment
like we do, please refrain from printing emails. It helps to keep the
environment forested and litter-free.
