Spark 1.5.0
data:
p1,lo1,8,0,4,0,5,20150901|50000,10000,1.0
p1,lo2,8,0,4,0,5,20150901|50000,10000,1.0
p1,lo3,8,0,4,0,5,20150901|50000,10000,1.0
p1,lo4,8,0,4,0,5,20150901|50000,10000,1.0
p1,lo1,8,0,4,0,5,20150901|50000,10000,1.0
p1,lo2,8,0,4,0,5,20150901|50000,10000,1.0
p1,lo3,8,0,4,0,5,20150901|50000,10000,1.0
p1,lo4,8,0,4,0,5,20150901|50000,10000,1.0
p1,lo1,8,0,4,0,5,20150901|50000,10000,1.0
p1,lo2,8,0,4,0,5,20150901|50000,10000,1.0
p1,lo3,8,0,4,0,5,20150901|50000,10000,1.0
p1,lo4,8,0,4,0,5,20150901|50000,10000,1.0
p1,lo1,8,0,4,0,5,20150901|50000,10000,1.0
p1,lo2,8,0,4,0,5,20150901|50000,10000,1.0
p1,lo3,8,0,4,0,5,20150901|50000,10000,1.0
p1,lo4,8,0,4,0,5,20150901|50000,10000,1.0
spark-shell:
spark-shell \
--num-executors 2 \
--driver-memory 1g \
--executor-memory 10g \
--executor-cores 8 \
--master yarn-client
case class Mykey(uname:String, lo:String, f1:Char, f2:Char, f3:Char,
f4:Char, f5:Char, f6:String)
case class Myvalue(count1:Long, count2:Long, num:Double)
val myrdd = sc.textFile("/user/al733a/mydata.txt").map { case line => {
val spl = line.split("\\|", -1)
val k = spl(0).split(",")
val v = spl(1).split(",")
(Mykey(k(0), k(1), k(2)(0).toChar, k(3)(0).toChar, k(4)(0).toChar,
k(5)(0).toChar, k(6)(0).toChar, k(7)),
Myvalue(v(0).toLong, v(1).toLong, v(2).toDouble)
)
}}
myrdd.groupByKey().map { case (mykey, val_iterable) => (mykey, 1)
}.collect().foreach(println)
(Mykey(p1,lo1,8,0,4,0,5,20150901),1)
(Mykey(p1,lo1,8,0,4,0,5,20150901),1)
(Mykey(p1,lo3,8,0,4,0,5,20150901),1)
(Mykey(p1,lo3,8,0,4,0,5,20150901),1)
(Mykey(p1,lo4,8,0,4,0,5,20150901),1)
(Mykey(p1,lo4,8,0,4,0,5,20150901),1)
(Mykey(p1,lo2,8,0,4,0,5,20150901),1)
(Mykey(p1,lo2,8,0,4,0,5,20150901),1)
You can see that each key is repeated 2 times but each key should only
appear once.
Arun
On Mon, Jan 4, 2016 at 4:07 PM, Ted Yu <[email protected]> wrote:
> Can you give a bit more information ?
>
> Release of Spark you're using
> Minimal dataset that shows the problem
>
> Cheers
>
> On Mon, Jan 4, 2016 at 3:55 PM, Arun Luthra <[email protected]> wrote:
>
>> I tried groupByKey and noticed that it did not group all values into the
>> same group.
>>
>> In my test dataset (a Pair rdd) I have 16 records, where there are only 4
>> distinct keys, so I expected there to be 4 records in the groupByKey
>> object, but instead there were 8. Each of the 4 distinct keys appear 2
>> times.
>>
>> Is this the expected behavior? I need to be able to get ALL values
>> associated with each key grouped into a SINGLE record. Is it possible?
>>
>> Arun
>>
>> p.s. reducebykey will not be sufficient for me
>>
>
>
