I see Javadoc Style documentation but nothing that looks like a code sample
I tried the following before asking
public static class LongAccumulableParam implements
AccumulableParam<Long,Long>,Serializable {
@Override
public Long addAccumulator(final Long r, final Long t) {
return r + t;
}
@Override
public Long addInPlace(final Long r1, final Long r2) {
return r1 + r2;
}
@Override
public Long zero(final Long initialValue) {
return 0L;
}
}
sparkContext.currentContext.accumulator(0L, "myAccumulator,
new LongAccumulableParam ());
does not compile which is why I ask for code samples
The Javados for accumulator says
*accumulator
<https://spark.apache.org/docs/1.0.0/api/java/org/apache/spark/api/java/JavaSparkContext.html#accumulator(T,
org.apache.spark.AccumulatorParam)>*(T initialValue, AccumulatorParam
<https://spark.apache.org/docs/1.0.0/api/java/org/apache/spark/AccumulatorParam.html>
<T> accumulatorParam)
Create an Accumulator
<https://spark.apache.org/docs/1.0.0/api/java/org/apache/spark/Accumulator.html>
variable
of a given type, which tasks can "add" values to using the add method.
which is a LONG way from a working code sample
On Wed, Nov 12, 2014 at 8:18 PM, Sean Owen <[email protected]> wrote:
> It's the exact same API you've already found, and it's documented:
>
> http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.AccumulatorParam
>
> JavaSparkContext has helper methods for int and double but not long. You
> can just make your own little implementation of AccumulatorParam<Long>
> right? ... which would be nice to add to JavaSparkContext.
>
> On Wed, Nov 12, 2014 at 11:05 PM, Steve Lewis <[email protected]>
> wrote:
>
>> JavaSparkContext currentContext = ...;
>> Accumulator<Integer> accumulator = currentContext.accumulator(0,
>> "MyAccumulator");
>>
>> will create an Accumulator of Integers. For many large Data problems
>> Integer is too small and Long is a better type.
>> I see a call like the following
>>
>> AccumulatorParam<Long> param = ?? how do I get one of these;
>> // no compiler is unhappy with this call
>> Accumulator<Long> accumulatorLong =
>> currentContext.accumulator(new Long(0), acc ,param);
>>
>> but NONE - ZERO documentation on its use, how to get a
>> AccumulatorParam<Long> or how to turn one into an Accumulator.
>> Any ideas
>>
>>
>>
>
--
Steven M. Lewis PhD
4221 105th Ave NE
Kirkland, WA 98033
206-384-1340 (cell)
Skype lordjoe_com
