Re: [VOTE] Release Apache Spark 2.3.3 (RC1)

2019-01-18 Thread Takeshi Yamamuro
Hi, sean, I run these tests again though, these tests passed on my AWS env. But, I notice these streaming tests are a little flaky though... On Sat, Jan 19, 2019 at 1:51 AM Sean Owen wrote: > The release itself looks OK. I'm getting, as before, a lot of errors > on the machine I'm building on.

Re: [VOTE] Release Apache Spark 2.3.3 (RC1)

2019-01-18 Thread Sean Owen
The release itself looks OK. I'm getting, as before, a lot of errors on the machine I'm building on. Is anyone else seeing this? if not I'm going to scrap the env and try something new. Errors like: - event ordering *** FAILED *** The code passed to failAfter did not complete within 10 seconds.

Re: Profile Spark Executors

2019-01-18 Thread Oleg Mazurov
statsd-jvm-profiler is a very basic profiler written in Java. It's little better than running jstack in a loop. From that perspective, it can't do real CPU profiling. What it presents as such is a collection of stack traces of Java threads in RUNNABLE state. RUNNABLE doesn't mean actually running o

Profile Spark Executors

2019-01-18 Thread Jack Kolokasis
Hi all,     I try to profile my spark executors performance when use on Heap persistent level in compare to use off-Heap persistent level. I use statsd-jvm-profiler to profile each executor. From the results i see that application spends 71,92% of its threads running the method sun.nio.ch.EP