What have you tried so far?
Maybe, the easiest way is using a collection and reduce them adding its
values.
JavaPairRDD<String, String> pairRDD = sc.parallelizePairs(data);
JavaPairRDD<String, List<Integer>> result = pairRDD.mapToPair(new
Functions.createList())
.mapToPair(new Functions.ListStringToInt())
.reduceByKey(new SumList());
Functions implementation using Java 7. Java 8 should be more simple.
public static final class SumList implements Function2<List<Integer>,
List<Integer>, List<Integer>> {
@Override public List<Integer> call(List<Integer> l1,
List<Integer> l2) throws Exception {
List<Integer> result = new ArrayList<Integer>();
for(int i=0;i<l1.size();++i){
result.add(l1.get(i)+l2.get(i));
}
return result;
}
}
public static final class ListStringToInt implements
PairFunction<Tuple2<String, List<String>>, String,
List<Integer>> {
@Override public Tuple2<String, List<Integer>> call(Tuple2<String,
List<String>> tuple2) throws Exception {
List<Integer> result = new ArrayList<Integer>();
for(String number : tuple2._2()){
result.add(Integer.valueOf(number));
}
return new Tuple2<String, List<Integer>>(tuple2._1(),result);
}
}
public static final class createList implements
PairFunction<Tuple2<String, String>, String,
List<String>> {
@Override public Tuple2<String, List<String>> call(Tuple2<String,
String> tuple2) throws Exception {
return new Tuple2<String, List<String>>(tuple2._1(),
Arrays.asList(tuple2._2().split(",")));
}
}
2015-05-14 15:40 GMT+02:00 Yasemin Kaya <[email protected]>:
> Hi,
>
> I have JavaPairRDD<String, String> and I want to implement reduceByKey
> method.
>
> My pairRDD :
> *2553: 0,0,0,1,0,0,0,0*
> 46551: 0,1,0,0,0,0,0,0
> 266: 0,1,0,0,0,0,0,0
> *2553: 0,0,0,0,0,1,0,0*
>
> *225546: 0,0,0,0,0,1,0,0*
> *225546: 0,0,0,0,0,1,0,0*
>
> I want to get :
> *2553: 0,0,0,1,0,1,0,0*
> 46551: 0,1,0,0,0,0,0,0
> 266: 0,1,0,0,0,0,0,0
> *225546: 0,0,0,0,0,2,0,0*
>
> Anyone can help me getting that?
> Thank you.
>
> Have a nice day.
> yasemin
>
> --
> hiç ender hiç
>
--
Gaspar Muñoz
@gmunozsoria
<http://www.stratio.com/>
Vía de las dos Castillas, 33, Ática 4, 3ª Planta
28224 Pozuelo de Alarcón, Madrid
Tel: +34 91 352 59 42 // *@stratiobd <https://twitter.com/StratioBD>*
