[
https://issues.apache.org/jira/browse/KUDU-3056?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041342#comment-17041342
]
ASF subversion and git services commented on KUDU-3056:
-------------------------------------------------------
Commit 5f199c16d46fc6bd6003f00d4e56ba75923f192c in kudu's branch
refs/heads/master from Grant Henke
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=5f199c1 ]
KUDU-3056: Reduce HdrHistogramAccumulator overhead
This patch makes a few changes to reduce the overhead of the
HdrHistogramAccumulator.
It changes from using `SynchronizedHistogram` (value type
long) to using `IntCountsHistogram` (value type int).
This significantly reduces the data footprint of the histogram and is
safe given write durations will never exceed `Integer.MAX_VALUE`.
Because thread safety is still important we syncronize all access
to `IntCountsHistogram` in `HistogramWrapper`.
It also adjusts the `HistogramWrapper` to lazily instantiate an
`IntCountsHistogram`. This means that if no values are recorded,
the overhead of the `HdrHistogramAccumulator` should be almost
zero.
Lastly it reduces the `numberOfSignificantValueDigits` tracked
in the histogram from 3 to 2. The result is relatively similar
output in the Spark accumulator with a significantly smaller
histogram.
I tested each variant using `getEstimatedFootprintInBytes()` and
the result is that the new implementation is 90% smaller when the
HdrHistogramAccumulator is used. The new implementation
is 100% smaller when no values are stored:
long w/ precision 3 & max 30000ms: 49664 (current)
long w/ precision 2 & max 30000ms: 9728
long w/ precision 1 & max 30000ms: 2048
int w/ precision 3 & max 30000ms: 25088
int w/ precision 2 & max 30000ms: 5120 (new)
int w/ precision 1 & max 30000ms: 1280
Note: I used a max of 30000ms in these calculations because that
is the default operation timeout
Below is sample string output from before and after this patch
generated with 1000 random values between 0ms and 500ms.
Before:
0.2%: 0ms, 50.3%: 265ms, 75.1%: 376ms, 87.5%: 437ms, 93.8%: 470ms, 96.9%:
484ms, 98.6%: 493ms, 99.5%: 496ms, 99.8%: 498ms, 100.0%: 499ms, 100.0%: 499ms
After:
0.2%: 0ms, 50.3%: 265ms, 75.4%: 377ms, 87.5%: 437ms, 93.9%: 471ms, 97.3%:
485ms, 98.6%: 493ms, 99.5%: 497ms, 100.0%: 499ms, 100.0%: 499ms
Note: I used the same seed so as to generate the same values for both strings.
Change-Id: Ic7c2a33bc61a2baa38703ea3340a07e06ab39db3
Reviewed-on: http://gerrit.cloudera.org:8080/15254
Reviewed-by: Adar Dembo <[email protected]>
Tested-by: Grant Henke <[email protected]>
> kudu-spark HdrHistogramAccumulator is too big, and make spark job failed
> --------------------------------------------------------------------------
>
> Key: KUDU-3056
> URL: https://issues.apache.org/jira/browse/KUDU-3056
> Project: Kudu
> Issue Type: Bug
> Components: spark
> Affects Versions: 1.9.0
> Reporter: caiconghui
> Assignee: Grant Henke
> Priority: Major
> Fix For: NA
>
> Attachments: heap1.png, heap2.png, heap3.png
>
> Original Estimate: 12h
> Remaining Estimate: 12h
>
> in production envrinment, we use kudu-spark to read kudu table, but even we
> don't use the
> HdrHistogramAccumulator, the HdrHistogramAccumulator stored in an array is
> stiil so big,
> totoal of them are almost 2 MB, so that when the number of kudu-spark
> task(for read kudu data and shuffle) is more than 900, the spark job failed,
> and the follwing error occured,
>
> Job aborted due to stage failure: Total size of serialized results of 1413
> tasks (3.0 GB) is bigger than spark.driver.maxResultSize (3.0 GB)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
