Hi,
Trying to read HDFS file with TextInputFormat.
scala> import org.apache.hadoop.mapred.TextInputFormat
scala> import org.apache.hadoop.io.{LongWritable, Text}
scala> val file2 =
sc.newAPIHadoopFile[LongWritable,Text,TextInputFormat]("hdfs://
192.168.100.130:8020/user/hue/pig/examples/data/sonnets.txt")
This is giving me the error.
<console>:14: error: type arguments
[org.apache.hadoop.io.LongWritable,org.apache.hadoop.io.Text,org.apache.hadoop.mapred.TextInputFormat]
conform to the bounds of none of the overloaded alternatives of
value newAPIHadoopFile: [K, V, F <:
org.apache.hadoop.mapreduce.InputFormat[K,V]](path: String, fClass:
Class[F], kClass: Class[K], vClass: Class[V], conf:
org.apache.hadoop.conf.Configuration)org.apache.spark.rdd.RDD[(K, V)] <and>
[K, V, F <: org.apache.hadoop.mapreduce.InputFormat[K,V]](path:
String)(implicit km: scala.reflect.ClassTag[K], implicit vm:
scala.reflect.ClassTag[V], implicit fm:
scala.reflect.ClassTag[F])org.apache.spark.rdd.RDD[(K, V)]
val file2 =
sc.newAPIHadoopFile[LongWritable,Text,TextInputFormat]("hdfs://
192.168.100.130:8020/user/hue/pig/examples/data/sonnets.txt")
What is correct syntax if I want to use TextInputFormat.
Also, how to use customInputFormat. Very silly question but I am not sure
how and where to keep jar file containing customInputFormat class.
Thanks
Pariksheet
--
Cheers,
Pari
