Rajat Kumer wrote: "Cannot find project Scala library 2.12.12 for module SparkSimpleApp"
So when I google this error message I find scala project maven sync failed <https://youtrack.jetbrains.com/issue/SCL-18941> søn. 27. feb. 2022 kl. 21:59 skrev Mich Talebzadeh < mich.talebza...@gmail.com>: > sorry which error? > > > > view my Linkedin profile > <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> > > > https://en.everybodywiki.com/Mich_Talebzadeh > > > > *Disclaimer:* Use it at your own risk. Any and all responsibility for any > loss, damage or destruction of data or any other property which may arise > from relying on this email's technical content is explicitly disclaimed. > The author will in no case be liable for any monetary damages arising from > such loss, damage or destruction. > > > > > On Sun, 27 Feb 2022 at 20:52, Bjørn Jørgensen <bjornjorgen...@gmail.com> > wrote: > >> Anyway I did google on your error and found this scala project maven >> sync failed <https://youtrack.jetbrains.com/issue/SCL-18941> >> Is this the same as the one you are getting? >> >> >> søn. 27. feb. 2022 kl. 21:16 skrev Mich Talebzadeh < >> mich.talebza...@gmail.com>: >> >>> Thanks Bjorn. I am aware of that. I just really wanted to create the >>> uber jar files with both sbt and maven in Intellij. >>> >>> >>> cheers >>> >>> >>> >>> view my Linkedin profile >>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>> >>> >>> https://en.everybodywiki.com/Mich_Talebzadeh >>> >>> >>> >>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>> any loss, damage or destruction of data or any other property which may >>> arise from relying on this email's technical content is explicitly >>> disclaimed. The author will in no case be liable for any monetary damages >>> arising from such loss, damage or destruction. >>> >>> >>> >>> >>> On Sun, 27 Feb 2022 at 20:12, Bjørn Jørgensen <bjornjorgen...@gmail.com> >>> wrote: >>> >>>> Mitch: You are using scala 2.11 to do this. Have a look at Building >>>> Spark <https://spark.apache.org/docs/latest/building-spark.html> "Spark >>>> requires Scala 2.12/2.13; support for Scala 2.11 was removed in Spark >>>> 3.0.0." >>>> >>>> søn. 27. feb. 2022 kl. 20:55 skrev Mich Talebzadeh < >>>> mich.talebza...@gmail.com>: >>>> >>>>> OK I decided to give a try to maven. >>>>> >>>>> Downloaded maven and unzipped the file WSL-Ubuntu terminal as unzip >>>>> apache-maven-3.8.4-bin.zip >>>>> >>>>> Then added to Windows env variable as MVN_HOME and added the bin >>>>> directory to path in windows. Restart intellij to pick up the correct >>>>> path. >>>>> >>>>> Again on the command line in intellij do >>>>> >>>>> *mvn -v* >>>>> Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537) >>>>> Maven home: d:\temp\apache-maven-3.8.4 >>>>> Java version: 1.8.0_73, vendor: Oracle Corporation, runtime: >>>>> C:\Program Files\Java\jdk1.8.0_73\jre >>>>> Default locale: en_GB, platform encoding: Cp1252 >>>>> OS name: "windows 10", version: "10.0", arch: "amd64", family: >>>>> "windows" >>>>> >>>>> in Intellij add maven support to your project. Follow this link Add >>>>> Maven support to an existing project >>>>> <https://www.jetbrains.com/help/idea/convert-a-regular-project-into-a-maven-project.html> >>>>> >>>>> There will be a pom.xml file under project directory >>>>> >>>>> [image: image.png] >>>>> >>>>> Edit that pom.xml file and add the following >>>>> >>>>> <?xml version="1.0" encoding="UTF-8"?> >>>>> <project xmlns="http://maven.apache.org/POM/4.0.0"; >>>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; >>>>> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 >>>>> http://maven.apache.org/xsd/maven-4.0.0.xsd";> >>>>> <modelVersion>4.0.0</modelVersion> >>>>> >>>>> <groupId>spark</groupId> >>>>> <artifactId>MichTest</artifactId> >>>>> <version>1.0</version> >>>>> >>>>> <properties> >>>>> <maven.compiler.source>8</maven.compiler.source> >>>>> <maven.compiler.target>8</maven.compiler.target> >>>>> </properties> >>>>> <dependencies> >>>>> <dependency> >>>>> <groupId>org.scala-lang</groupId> >>>>> <artifactId>scala-library</artifactId> >>>>> <version>2.11.7</version> >>>>> </dependency> >>>>> <dependency> >>>>> <groupId>org.apache.spark</groupId> >>>>> <artifactId>spark-core_2.10</artifactId> >>>>> <version>2.0.0</version> >>>>> </dependency> >>>>> <dependency> >>>>> <groupId>org.apache.spark</groupId> >>>>> <artifactId>spark-sql_2.10</artifactId> >>>>> <version>2.0.0</version> >>>>> </dependency> >>>>> </dependencies> >>>>> </project> >>>>> >>>>> In intellij open a Terminal under project sub-directory where the pom >>>>> file is created and you edited. >>>>> >>>>> >>>>> *mvn clean* >>>>> >>>>> [INFO] Scanning for projects... >>>>> >>>>> [INFO] >>>>> >>>>> [INFO] ---------------------------< spark:MichTest >>>>> >--------------------------- >>>>> >>>>> [INFO] Building MichTest 1.0 >>>>> >>>>> [INFO] --------------------------------[ jar >>>>> ]--------------------------------- >>>>> >>>>> [INFO] >>>>> >>>>> [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ MichTest --- >>>>> >>>>> [INFO] Deleting D:\temp\intellij\MichTest\target >>>>> >>>>> [INFO] >>>>> ------------------------------------------------------------------------ >>>>> >>>>> [INFO] BUILD SUCCESS >>>>> >>>>> [INFO] >>>>> ------------------------------------------------------------------------ >>>>> >>>>> [INFO] Total time: 4.451 s >>>>> >>>>> [INFO] Finished at: 2022-02-27T19:37:57Z >>>>> >>>>> [INFO] >>>>> ------------------------------------------------------------------------ >>>>> >>>>> *mvn compile* >>>>> >>>>> [INFO] Scanning for projects... >>>>> >>>>> [INFO] >>>>> >>>>> [INFO] ---------------------------< spark:MichTest >>>>> >--------------------------- >>>>> >>>>> [INFO] Building MichTest 1.0 >>>>> >>>>> [INFO] --------------------------------[ jar >>>>> ]--------------------------------- >>>>> >>>>> [INFO] >>>>> >>>>> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ >>>>> MichTest --- >>>>> >>>>> [WARNING] Using platform encoding (Cp1252 actually) to copy filtered >>>>> resources, i.e. build is platform dependent! >>>>> >>>>> [INFO] Copying 0 resource >>>>> >>>>> [INFO] >>>>> >>>>> [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ >>>>> MichTest --- >>>>> >>>>> [INFO] Nothing to compile - all classes are up to date >>>>> >>>>> [INFO] >>>>> ------------------------------------------------------------------------ >>>>> >>>>> [INFO] BUILD SUCCESS >>>>> >>>>> [INFO] >>>>> ------------------------------------------------------------------------ >>>>> >>>>> [INFO] Total time: 1.242 s >>>>> >>>>> [INFO] Finished at: 2022-02-27T19:38:58Z >>>>> >>>>> [INFO] >>>>> ------------------------------------------------------------------------ >>>>> >>>>> Now create the package >>>>> >>>>> *mvn package* >>>>> [INFO] Scanning for projects... >>>>> [INFO] >>>>> [INFO] ---------------------------< spark:MichTest >>>>> >--------------------------- >>>>> [INFO] Building MichTest 1.0 >>>>> [INFO] --------------------------------[ jar >>>>> ]--------------------------------- >>>>> [INFO] >>>>> [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ >>>>> MichTest --- >>>>> [WARNING] Using platform encoding (Cp1252 actually) to copy filtered >>>>> resources, i.e. build is platform dependent! >>>>> [INFO] Copying 0 resource >>>>> [INFO] >>>>> [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ >>>>> MichTest --- >>>>> [INFO] Nothing to compile - all classes are up to date >>>>> [INFO] >>>>> [INFO] --- maven-resources-plugin:2.6:testResources >>>>> (default-testResources) @ MichTest --- >>>>> [WARNING] Using platform encoding (Cp1252 actually) to copy filtered >>>>> resources, i.e. build is platform dependent! >>>>> [INFO] skip non existing resourceDirectory >>>>> D:\temp\intellij\MichTest\src\test\resources >>>>> [INFO] >>>>> [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) >>>>> @ MichTest --- >>>>> [INFO] Nothing to compile - all classes are up to date >>>>> [INFO] >>>>> [INFO] --- maven-surefire-plugin:2.12.4:test (default-test) @ MichTest >>>>> --- >>>>> [INFO] No tests to run. >>>>> [INFO] >>>>> [INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ MichTest --- >>>>> [INFO] *Building jar: >>>>> D:\temp\intellij\MichTest\target\scala-2.11\MichTest-1.0.jar* >>>>> [INFO] >>>>> ------------------------------------------------------------------------ >>>>> [INFO] BUILD SUCCESS >>>>> [INFO] >>>>> ------------------------------------------------------------------------ >>>>> [INFO] Total time: 1.511 s >>>>> [INFO] Finished at: 2022-02-27T19:40:22Z >>>>> [INFO] >>>>> ------------------------------------------------------------------------ >>>>> >>>>> sbt should work from the same directory as well. >>>>> >>>>> I find it easier* to package from the command line in Intellij >>>>> Terminal. * >>>>> >>>>> Let me know if this is a wrong approach or if there are better ways of >>>>> doing it. >>>>> >>>>> HTH >>>>> >>>>> >>>>> view my Linkedin profile >>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>> >>>>> >>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>> >>>>> >>>>> >>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility for >>>>> any loss, damage or destruction of data or any other property which may >>>>> arise from relying on this email's technical content is explicitly >>>>> disclaimed. The author will in no case be liable for any monetary damages >>>>> arising from such loss, damage or destruction. >>>>> >>>>> >>>>> >>>>> >>>>> On Sun, 27 Feb 2022 at 10:05, Mich Talebzadeh < >>>>> mich.talebza...@gmail.com> wrote: >>>>> >>>>>> Got curious with this intellij stuff. >>>>>> >>>>>> I recall using sbt rather than MVN so go to terminal in your intellij >>>>>> and verify what is installed >>>>>> >>>>>> sbt -version >>>>>> sbt version in this project: 1.3.4 >>>>>> sbt script version: 1.3.4 >>>>>> >>>>>> scala -version >>>>>> >>>>>> Scala code runner version 2.11.7 -- Copyright 2002-2013, LAMP/EPFL >>>>>> >>>>>> java --version >>>>>> openjdk 11.0.7 2020-04-14 >>>>>> OpenJDK Runtime Environment AdoptOpenJDK (build 11.0.7+10) >>>>>> OpenJDK 64-Bit Server VM AdoptOpenJDK (build 11.0.7+10, mixed mode) >>>>>> >>>>>> For now in the directory where you have Main.scala create build.sbt >>>>>> file >>>>>> >>>>>> // The simplest possible sbt build file is just one line: >>>>>> >>>>>> scalaVersion := "2.11.7" >>>>>> // That is, to create a valid sbt build, all you've got to do is define >>>>>> the >>>>>> // version of Scala you'd like your project to use. >>>>>> >>>>>> / To learn more about multi-project builds, head over to the official sbt >>>>>> // documentation at http://www.scala-sbt.org/documentation.html >>>>>> libraryDependencies += "org.scala-lang.modules" %% >>>>>> "scala-parser-combinators" % "1.1.2" >>>>>> libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.1.0" >>>>>> >>>>>> >>>>>> This is my Main.scala file a copy from sparkbyexample >>>>>> <https://sparkbyexamples.com/spark/spark-setup-run-with-scala-intellij/> >>>>>> >>>>>> >>>>>> package org.example >>>>>> import org.apache.spark.sql.SparkSession >>>>>> object SparkSessionTest extends App{ >>>>>> val spark = SparkSession.builder() >>>>>> .master("local[1]") >>>>>> .appName("SparkByExample") >>>>>> .getOrCreate(); >>>>>> >>>>>> println("First SparkContext:") >>>>>> println("APP Name :"+spark.sparkContext.appName); >>>>>> println("Deploy Mode :"+spark.sparkContext.deployMode); >>>>>> println("Master :"+spark.sparkContext.master); >>>>>> >>>>>> val sparkSession2 = SparkSession.builder() >>>>>> .master("local[1]") >>>>>> .appName("SparkByExample-test") >>>>>> .getOrCreate(); >>>>>> >>>>>> println("Second SparkContext:") >>>>>> println("APP Name :"+sparkSession2.sparkContext.appName); >>>>>> println("Deploy Mode :"+sparkSession2.sparkContext.deployMode); >>>>>> println("Master :"+sparkSession2.sparkContext.master); >>>>>> } >>>>>> >>>>>> Go back to Terminal under directory where you have both files >>>>>> built.sbt and Main.scala >>>>>> >>>>>> >>>>>> *sbt clean* >>>>>> >>>>>> [info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins >>>>>> >>>>>> [info] Loading project definition from >>>>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project >>>>>> >>>>>> [info] Loading settings for project scala from build.sbt ... >>>>>> >>>>>> [info] Set current project to MichTest (in build >>>>>> file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/) >>>>>> >>>>>> [success] Total time: 0 s, completed Feb 27, 2022 9:54:10 AM >>>>>> >>>>>> *sbt compile* >>>>>> [info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins >>>>>> [info] Loading project definition from >>>>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project >>>>>> [info] Loading settings for project scala from build.sbt ... >>>>>> [info] Set current project to MichTest (in build >>>>>> file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/) >>>>>> [info] Executing in batch mode. For better performance use sbt's shell >>>>>> [warn] There may be incompatibilities among your library >>>>>> dependencies; run 'evicted' to see detailed eviction warnings. >>>>>> [info] Compiling 1 Scala source to >>>>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\target\scala-2.11\classes >>>>>> ... >>>>>> [success] Total time: 5 s, completed Feb 27, 2022 9:55:10 AM >>>>>> >>>>>> *sbt package* >>>>>> >>>>>> [info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins >>>>>> >>>>>> [info] Loading project definition from >>>>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\project >>>>>> >>>>>> [info] Loading settings for project scala from build.sbt ... >>>>>> >>>>>> [info] Set current project to MichTest (in build >>>>>> file:/D:/temp/intellij/MichTest/src/main/scala/com/ctp/training/scala/) >>>>>> >>>>>> [success] Total time: 1 s, completed Feb 27, 2022 9:56:48 AM >>>>>> >>>>>> *ls* >>>>>> >>>>>> >>>>>> Directory: >>>>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala >>>>>> >>>>>> >>>>>> Mode LastWriteTime Length Name >>>>>> >>>>>> ---- ------------- ------ ---- >>>>>> >>>>>> d----- 2/27/2022 7:04 AM null >>>>>> >>>>>> d----- 2/27/2022 8:33 AM project >>>>>> >>>>>> d----- 2/27/2022 9:08 AM spark-warehouse >>>>>> >>>>>> d----- 2/27/2022 9:55 AM target >>>>>> >>>>>> -a---- 2/27/2022 9:17 AM 3511 build.sbt >>>>>> >>>>>> Note that you have target directory and underneath scala-2.11 (in my >>>>>> case) and the uber jar file michtest_2.11-1.0.jar >>>>>> >>>>>> *ls* >>>>>> Directory: >>>>>> D:\temp\intellij\MichTest\src\main\scala\com\ctp\training\scala\target\scala-2.11 >>>>>> Mode LastWriteTime Length Name >>>>>> ---- ------------- ------ ---- >>>>>> d----- 2/27/2022 9:55 AM classes >>>>>> d----- 2/27/2022 9:55 AM update >>>>>> -a---- 2/27/2022 9:56 AM 3938 >>>>>> *michtest_2.11-1.0.jar* >>>>>> >>>>>> >>>>>> These are old stuff but still shows how to create a jar file with sbt >>>>>> >>>>>> HTH >>>>>> >>>>>> >>>>>> view my Linkedin profile >>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>> >>>>>> >>>>>> >>>>>> https://en.everybodywiki.com/Mich_Talebzadeh >>>>>> >>>>>> >>>>>> >>>>>> *Disclaimer:* Use it at your own risk. Any and all responsibility >>>>>> for any loss, damage or destruction of data or any other property which >>>>>> may >>>>>> arise from relying on this email's technical content is explicitly >>>>>> disclaimed. The author will in no case be liable for any monetary damages >>>>>> arising from such loss, damage or destruction. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Sat, 26 Feb 2022 at 22:48, Sean Owen <sro...@gmail.com> wrote: >>>>>> >>>>>>> I don't think any of that is related, no. >>>>>>> How are you dependencies set up? manually with IJ, or in a build >>>>>>> file (Maven, Gradle)? Normally you do the latter and dependencies are >>>>>>> taken >>>>>>> care of for you, but you app would definitely have to express a >>>>>>> dependency >>>>>>> on Scala libs. >>>>>>> >>>>>>> On Sat, Feb 26, 2022 at 4:25 PM Bitfox <bit...@bitfox.top> wrote: >>>>>>> >>>>>>>> Java SDK installed? >>>>>>>> >>>>>>>> On Sun, Feb 27, 2022 at 5:39 AM Sachit Murarka < >>>>>>>> connectsac...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Hello , >>>>>>>>> >>>>>>>>> Thanks for replying. I have installed Scala plugin in IntelliJ >>>>>>>>> first then also it's giving same error >>>>>>>>> >>>>>>>>> Cannot find project Scala library 2.12.12 for module SparkSimpleApp >>>>>>>>> >>>>>>>>> Thanks >>>>>>>>> Rajat >>>>>>>>> >>>>>>>>> On Sun, Feb 27, 2022, 00:52 Bitfox <bit...@bitfox.top> wrote: >>>>>>>>> >>>>>>>>>> You need to install scala first, the current version for spark is >>>>>>>>>> 2.12.15 >>>>>>>>>> I would suggest you install scala by sdk which works great. >>>>>>>>>> >>>>>>>>>> Thanks >>>>>>>>>> >>>>>>>>>> On Sun, Feb 27, 2022 at 12:10 AM rajat kumar < >>>>>>>>>> kumar.rajat20...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Hello Users, >>>>>>>>>>> >>>>>>>>>>> I am trying to create spark application using Scala(Intellij). >>>>>>>>>>> I have installed Scala plugin in intelliJ still getting below >>>>>>>>>>> error:- >>>>>>>>>>> >>>>>>>>>>> Cannot find project Scala library 2.12.12 for module >>>>>>>>>>> SparkSimpleApp >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Could anyone please help what I am doing wrong? >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> >>>>>>>>>>> Rajat >>>>>>>>>>> >>>>>>>>>> >>>> >>>> -- >>>> Bjørn Jørgensen >>>> Vestre Aspehaug 4, 6010 Ålesund >>>> Norge >>>> >>>> +47 480 94 297 >>>> >>> >> >> -- >> Bjørn Jørgensen >> Vestre Aspehaug 4, 6010 Ålesund >> Norge >> >> +47 480 94 297 >> > -- Bjørn Jørgensen Vestre Aspehaug 4, 6010 Ålesund Norge +47 480 94 297
