Perhaps you can rewrite the functions in terms of other kernels that can be merged -- for example something like the following
stddev(x) = sqrt((sum(x*x) - sum(x)*sum(x) / count(x))/(count(x)-1))) (loosely translated from https://math.stackexchange.com/questions/102978/incremental-computation-of-standard-deviation ) On Wed, Sep 16, 2020 at 6:12 AM Yibo Cai <yibo....@arm.com> wrote: > Hi, > > I have a question about aggregate kernel implementation. Any help is > appreciated. > > Aggregate kernel implements "consume" and "merge" interfaces. For a > chunked array, "consume" is called for each array to get a temporary > aggregated result, then "merge" it with previously consumed result. For > associative operations like min/max/sum, this pattern is convenient. We can > easily "merge" min/max/sum of two arrays, e.g, sum([array_a, array_b]) = > sum(array_a) + sum(array_b). > > But I wonder what's the best approach to deal with operations like > stdev/percentile. Results of these operations cannot be easily "merged". We > have to walk through all the chunks to get the result. For these > operations, looks "consume" must copy the input array and do all > calculation once at "finalize" time. Or we don't expect it to support > chunked array for them. > > Yibo >
