Yes I've seen that error in the past too, and was just talking to Imran the other day about it. I thought it only occurred when the hive module was enabled, which I don't enable.
The problem is that the plugin that causes an error in IntelliJ for scalac is what parses these values.* I think he got it to work with this change: http://stackoverflow.com/questions/26788367/quasiquotes-in-intellij-14/26908554#26908554 If that works for you let's put it on the wiki. * probably an ignorant question but is this feature important enough to warrant the extra scala compiler plugin? the quasiquotes syntax I mean. On Sat, Jan 17, 2015 at 10:29 AM, Chunnan Yao <yaochun...@gmail.com> wrote: > *I followed the procedures instructed by > https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark#ContributingtoSpark-IntelliJ. > But problems still occurs which has made me a little bit annoyed. > > My environment settings are:JAVA 1.7.0 Scala: 2.10.4 Spark:1.2.0, Intellij > Idea 14.0.2, Ubuntu 14.04 > > Firstly I got the scala plugin correctly installed. > > I choosed maven-3, hadoop-2.4, scala-2.10 as my profiles when importing the > project. > > After importing, I first turned on "View-Tool Windows-Maven Projects". I see > the "hbase-hadoop1" is selected, but I had not chosen it in the import > process. So I deselected it to leave the hadoop-2.4, maven-3, scala-2.10 to > be the only three selected items in "Maven Projects-Profiles". > > According to the Wiki, the next step should be "Generate Sources and Update > Folders For All Projects". I did so, and waited for some minutes to get the > sub-projects prepared. > > Then I cleared the "Additional compiler options" in the > "File-Settings-Build, Execution, Deployment-Compiler-Scala Compiler". > > Finally I choosed Build-reBuild project. > > However, the compiler failed with "value q is not a member of stringcontext" > errors. * > > (partial screen shot) > ------------------------------------------------------- > /home/yaochunnan/workspace/spark_source/spark-1.2.0/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/GenerateProjection.scala > Error:(42, 21) value q is not a member of StringContext > val lengthDef = q"final val length = $tupleLength" > ^ > Error:(54, 7) value q is not a member of StringContext > q""" > ^ > Error:(66, 9) value q is not a member of StringContext > q""" > ^ > Error:(83, 9) value q is not a member of StringContext > q"if(isNullAt($iLit)) { null } else { ${newTermName(s"c$i")} }" > ^ > Error:(85, 7) value q is not a member of StringContext > q"override def iterator = Iterator[Any](..$allColumns)" > ^ > Error:(88, 27) value q is not a member of StringContext > val accessorFailure = q"""scala.sys.error("Invalid ordinal:" + i)""" > ^ > Error:(95, 9) value q is not a member of StringContext > q"if(i == $ordinal) { if(isNullAt($i)) return null else return > $elementName }" > ^ > Error:(97, 7) value q is not a member of StringContext > q"override def apply(i: Int): Any = { ..$cases; $accessorFailure }" > ^ > Error:(106, 9) value q is not a member of StringContext > q""" > ^ > Error:(117, 7) value q is not a member of StringContext > q"override def update(i: Int, value: Any): Unit = { ..$cases; > $accessorFailure }" > ^ > Error:(126, 11) value q is not a member of StringContext > q"if(i == $i) return $elementName" :: Nil > ^ > Error:(130, 7) value q is not a member of StringContext > q""" > ^ > Error:(143, 11) value q is not a member of StringContext > q"if(i == $i) { nullBits($i) = false; $elementName = value; return > }" :: Nil > ^ > Error:(147, 7) value q is not a member of StringContext > q""" > ^ > Error:(157, 29) value q is not a member of StringContext > case BooleanType => q"if ($elementName) 0 else 1" > ^ > Error:(158, 52) value q is not a member of StringContext > case ByteType | ShortType | IntegerType => q"$elementName.toInt" > ^ > Error:(159, 26) value q is not a member of StringContext > case LongType => q"($elementName ^ ($elementName >>> 32)).toInt" > ^ > Error:(160, 27) value q is not a member of StringContext > case FloatType => q"java.lang.Float.floatToIntBits($elementName)" > ^ > Error:(162, 11) value q is not a member of StringContext > q"{ val b = java.lang.Double.doubleToLongBits($elementName); (b ^ > (b >>>32)).toInt }" > ^ > Error:(163, 19) value q is not a member of StringContext > case _ => q"$elementName.hashCode" > ^ > Error:(165, 7) value q is not a member of StringContext > q"if (isNullAt($i)) 0 else $nonNull" > ^ > Error:(168, 54) value q is not a member of StringContext > val hashUpdates: Seq[Tree] = hashValues.map(v => q"""result = 37 * > result + $v""": Tree) > ^ > Error:(171, 7) value q is not a member of StringContext > q""" > ^ > Error:(181, 7) value q is not a member of StringContext > q"if (this.$elementName != specificType.$elementName) return false" > ^ > Error:(185, 7) value q is not a member of StringContext > q""" > ^ > Error:(195, 7) value q is not a member of StringContext > q""" > ^ > Error:(210, 16) value q is not a member of StringContext > val code = q""" > ^ > --------------------------------------------------------------------- > * > Then I tried > http://stackoverflow.com/questions/26995023/errorscalac-bad-option-p-intellij-idea, > which does not clear the "Additional compiler options", but change the -P in > to -Xplugin. > So now my "Additional Compiler Options" is like this > "-Xplugin:/home/yaochunnan/.m2/repository/org/scalamacros/paradise_2.10.4/2.0.1/paradise_2.10.4-2.0.1.jar" > > Then I reBuild again with the following errors: * > (partial screen shot) > ---------------------------------------------------------------------- > /home/yaochunnan/workspace/spark_source/spark-1.2.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveContext.scala > Error:(169, 38) not found: value HiveShim > Option(tableParameters.get(HiveShim.getStatsSetupConstTotalSize)) > ^ > Error:(177, 31) not found: value HiveShim > tableParameters.put(HiveShim.getStatsSetupConstTotalSize, > newTotalSize.toString) > ^ > Error:(292, 36) not found: value HiveShim > val proc: CommandProcessor = > HiveShim.getCommandProcessor(Array(tokens(0)), hiveconf) > ^ > Error:(304, 25) not found: value HiveShim > val results = HiveShim.createDriverResultsArray > ^ > Error:(314, 11) not found: value HiveShim > HiveShim.processResults(results) > ^ > Error:(418, 7) not found: value HiveShim > HiveShim.createDecimal(decimal.toBigDecimal.underlying()).toString > ^ > Error:(420, 7) not found: value HiveShim > HiveShim.createDecimal(decimal.underlying()).toString > ^ > /home/yaochunnan/workspace/spark_source/spark-1.2.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveInspectors.scala > Error:(97, 7) not found: value HiveShim > HiveShim.toCatalystDecimal( > ^ > Error:(123, 46) not found: value HiveShim > case hdoi: HiveDecimalObjectInspector => > HiveShim.toCatalystDecimal(hdoi, data) > ^ > Error:(156, 19) not found: value HiveShim > (o: Any) => > HiveShim.createDecimal(o.asInstanceOf[Decimal].toBigDecimal.underlying()) > ^ > Error:(210, 31) not found: value HiveShim > case b: BigDecimal => HiveShim.createDecimal(b.underlying()) > ^ > Error:(211, 28) not found: value HiveShim > case d: Decimal => > HiveShim.createDecimal(d.toBigDecimal.underlying()) > ^ > Error:(283, 7) not found: value HiveShim > HiveShim.getStringWritableConstantObjectInspector(value) > ^ > Error:(285, 7) not found: value HiveShim > HiveShim.getIntWritableConstantObjectInspector(value) > ^ > Error:(287, 7) not found: value HiveShim > HiveShim.getDoubleWritableConstantObjectInspector(value) > ^ > Error:(289, 7) not found: value HiveShim > HiveShim.getBooleanWritableConstantObjectInspector(value) > ^ > Error:(291, 7) not found: value HiveShim > HiveShim.getLongWritableConstantObjectInspector(value) > ^ > Error:(293, 7) not found: value HiveShim > HiveShim.getFloatWritableConstantObjectInspector(value) > ^ > Error:(295, 7) not found: value HiveShim > HiveShim.getShortWritableConstantObjectInspector(value) > ^ > Error:(297, 7) not found: value HiveShim > HiveShim.getByteWritableConstantObjectInspector(value) > ^ > Error:(299, 7) not found: value HiveShim > HiveShim.getBinaryWritableConstantObjectInspector(value) > ^ > Error:(301, 7) not found: value HiveShim > HiveShim.getDateWritableConstantObjectInspector(value) > ^ > Error:(303, 7) not found: value HiveShim > HiveShim.getTimestampWritableConstantObjectInspector(value) > ^ > Error:(305, 7) not found: value HiveShim > HiveShim.getDecimalWritableConstantObjectInspector(value) > ^ > Error:(307, 7) not found: value HiveShim > HiveShim.getPrimitiveNullWritableConstantObjectInspector > ^ > Error:(363, 51) not found: value HiveShim > case w: WritableHiveDecimalObjectInspector => > HiveShim.decimalTypeInfoToCatalyst(w) > ^ > Error:(364, 47) not found: value HiveShim > case j: JavaHiveDecimalObjectInspector => > HiveShim.decimalTypeInfoToCatalyst(j) > ^ > Error:(393, 30) not found: value HiveShim > case d: DecimalType => HiveShim.decimalTypeInfo(d) > ^ > /home/yaochunnan/workspace/spark_source/spark-1.2.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveMetastoreCatalog.scala > Error:(78, 11) not found: value HiveShim > HiveShim.getAllPartitionsOf(client, table).toSeq > ^ > Error:(205, 7) not found: value HiveShim > HiveShim.setLocation(tbl, crtTbl) > ^ > Error:(443, 28) not found: value HiveShim > case d: DecimalType => HiveShim.decimalMetastoreString(d) > ^ > Error:(472, 53) not found: value HiveShim > val totalSize = > hiveQlTable.getParameters.get(HiveShim.getStatsSetupConstTotalSize) > ^ > Error:(490, 19) not found: value HiveShim > val tableDesc = HiveShim.getTableDesc( > ^ > /home/yaochunnan/workspace/spark_source/spark-1.2.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveUdfs.scala > Error:(255, 18) not found: type HiveFunctionWrapper > funcWrapper: HiveFunctionWrapper, > ^ > Error:(73, 53) not found: type HiveFunctionWrapper > private[hive] case class HiveSimpleUdf(funcWrapper: HiveFunctionWrapper, > children: Seq[Expression]) > ^ > Error:(57, 25) not found: type HiveFunctionWrapper > HiveSimpleUdf(new HiveFunctionWrapper(functionClassName), children) > ^ > Error:(134, 54) not found: type HiveFunctionWrapper > private[hive] case class HiveGenericUdf(funcWrapper: HiveFunctionWrapper, > children: Seq[Expression]) > ^ > Error:(59, 26) not found: type HiveFunctionWrapper > HiveGenericUdf(new HiveFunctionWrapper(functionClassName), children) > ^ > Error:(185, 18) not found: type HiveFunctionWrapper > funcWrapper: HiveFunctionWrapper, > ^ > Error:(62, 27) not found: type HiveFunctionWrapper > HiveGenericUdaf(new HiveFunctionWrapper(functionClassName), children) > ^ > Error:(214, 18) not found: type HiveFunctionWrapper > funcWrapper: HiveFunctionWrapper, > ^ > Error:(64, 20) not found: type HiveFunctionWrapper > HiveUdaf(new HiveFunctionWrapper(functionClassName), children) > ^ > Error:(66, 27) not found: type HiveFunctionWrapper > HiveGenericUdtf(new HiveFunctionWrapper(functionClassName), Nil, > children) > ^ > Error:(322, 18) not found: type HiveFunctionWrapper > funcWrapper: HiveFunctionWrapper, > ^ > /home/yaochunnan/workspace/spark_source/spark-1.2.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala > Error:(1132, 15) not found: type HiveFunctionWrapper > new HiveFunctionWrapper(functionName), > ^ > /home/yaochunnan/workspace/spark_source/spark-1.2.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/InsertIntoHiveTable.scala > Error:(44, 34) object HiveShim is not a member of package > org.apache.spark.sql.hive > import org.apache.spark.sql.hive.HiveShim._ > ^ > Error:(43, 8) object ShimFileSinkDesc is not a member of package > org.apache.spark.sql.hive > import org.apache.spark.sql.hive.{ ShimFileSinkDesc => FileSinkDesc} > ^ > Error:(76, 21) not found: type FileSinkDesc > fileSinkConf: FileSinkDesc, > ^ > Error:(142, 23) not found: value HiveShim > val tmpLocation = HiveShim.getExternalTmpPath(hiveContext, > tableLocation) > ^ > Error:(143, 28) not found: type FileSinkDesc > val fileSinkConf = new FileSinkDesc(tmpLocation.toString, tableDesc, > false) > ^ > /home/yaochunnan/workspace/spark_source/spark-1.2.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/TableReader.scala > Error:(141, 22) not found: value HiveShim > val partPath = HiveShim.getDataLocationPath(partition) > ^ > Error:(298, 33) not found: value HiveShim > row.update(ordinal, HiveShim.toCatalystDecimal(oi, value)) > ^ > /home/yaochunnan/workspace/spark_source/spark-1.2.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/TestHive.scala > Error:(384, 3) not found: value HiveShim > HiveShim.createDefaultDBIfNeeded(this) > ^ > /home/yaochunnan/workspace/spark_source/spark-1.2.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/DescribeHiveTableCommand.scala > Error:(29, 8) object HiveShim is not a member of package > org.apache.spark.sql.hive > import org.apache.spark.sql.hive.HiveShim > ^ > /home/yaochunnan/workspace/spark_source/spark-1.2.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/HiveTableScan.scala > Error:(89, 5) not found: value HiveShim > HiveShim.appendReadColumns(hiveConf, neededColumnIDs, > attributes.map(_.name)) > ^ > /home/yaochunnan/workspace/spark_source/spark-1.2.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/hiveWriterContainers.scala > Error:(38, 34) object HiveShim is not a member of package > org.apache.spark.sql.hive > import org.apache.spark.sql.hive.HiveShim._ > ^ > Error:(37, 8) object ShimFileSinkDesc is not a member of package > org.apache.spark.sql.hive > import org.apache.spark.sql.hive.{ShimFileSinkDesc => FileSinkDesc} > ^ > Error:(174, 19) not found: type FileSinkDesc > fileSinkConf: FileSinkDesc, > ^ > Error:(46, 19) not found: type FileSinkDesc > fileSinkConf: FileSinkDesc) > ^ > Error:(220, 33) not found: type FileSinkDesc > val newFileSinkDesc = new FileSinkDesc( > ^ > /home/yaochunnan/workspace/spark_source/spark-1.2.0/sql/hive/src/main/scala/org/apache/spark/sql/hive/parquet/FakeParquetSerDe.scala > Warning:(34, 2) @deprecated now takes two arguments; see the scaladoc. > @deprecated("No code should depend on FakeParquetHiveSerDe as it is only > intended as a " + > ^ > ------------------------------------------------------------------------ > > *I thought it was the problem from Maven Profiles. So I tried reselecting > hbase-hadoop1 or hive or hbase-hadoop2. The error still occurs. Please help > me. This has annoyed me for a whole afternoon!* > > > > > ----- > Feel the sparking Spark! > -- > View this message in context: > http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-development-with-IntelliJ-tp10032p10163.html > Sent from the Apache Spark Developers List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
