Thanks Mark, far more interesting. I've improved the formatting to make it
easier to read (and fixed width font for email reading)
* Can you do same run with say 10 iterations of Jacobi PC?
* PCApply performance (looks like GAMG) is terrible! Problems too small?
* VecScatter time is completely dominated by SFPack! Junchao what's up with
that? Lots of little kernels in the PCApply? PCJACOBI run will help clarify
where that is coming from.
------------------------------------------------------------------------------------------------------------------------
Event Count Time (sec) Flop
--- Global --- --- Stage ---- Total GPU - CpuToGpu - - GpuToCpu - GPU
Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct
%T %F %M %L %R %T %F %M %L %R Mflop/s Mflop/s Count Size Count Size %F
---------------------------------------------------------------------------------------------------------------------------------------------------------------
MatMult 200 1.0 6.7831e-01 1.0 4.91e+10 1.0 1.1e+04 6.6e+04
1.0e+00 9 92 99 79 0 71 92100100 0 579,635 1,014,212 1 2.04e-04 0
0.00e+00 100
KSPSolve 1 1.0 9.4550e-01 1.0 5.31e+10 1.0 1.1e+04 6.6e+04
6.0e+02 12100 99 79 94 100100100100100 449,667 893,741 1 2.04e-04 0
0.00e+00 100
PCApply 201 1.0 1.6966e-01 1.0 3.09e+08 1.0 0.0e+00 0.0e+00
2.0e+00 2 1 0 0 0 18 1 0 0 0 14,558 16,3941 0 0.00e+00 0
0.00e+00 100
VecTDot 401 1.0 5.3642e-02 1.3 1.23e+09 1.0 0.0e+00 0.0e+00
4.0e+02 1 2 0 0 62 5 2 0 0 66 183,716 353,914 0 0.00e+00 0
0.00e+00 100
VecNorm 201 1.0 2.2219e-02 1.1 6.17e+08 1.0 0.0e+00 0.0e+00
2.0e+02 0 1 0 0 31 2 1 0 0 33 222,325 303,155 0 0.00e+00 0
0.00e+00 100
VecAXPY 400 1.0 2.3017e-02 1.1 1.23e+09 1.0 0.0e+00 0.0e+00
0.0e+00 0 2 0 0 0 2 2 0 0 0 427,091 514,744 0 0.00e+00 0
0.00e+00 100
VecAYPX 199 1.0 1.1312e-02 1.1 6.11e+08 1.0 0.0e+00 0.0e+00
0.0e+00 0 1 0 0 0 1 1 0 0 0 432,323 532,889 0 0.00e+00 0
0.00e+00 100
VecPointwiseMult 201 1.0 1.0471e-02 1.1 3.09e+08 1.0 0.0e+00 0.0e+00
0.0e+00 0 1 0 0 0 1 1 0 0 0 235,882 290,088 0 0.00e+00 0
0.00e+00 100
VecScatterBegin 200 1.0 1.8458e-01 1.1 0.00e+00 0.0 1.1e+04 6.6e+04
1.0e+00 2 0 99 79 0 19 0100100 0 0 0 1 2.04e-04 0
0.00e+00 0
VecScatterEnd 200 1.0 1.9007e-02 3.7 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 1 0 0 0 0 0 0 0 0.00e+00 0
0.00e+00 0
SFPack 200 1.0 1.7309e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00 2 0 0 0 0 18 0 0 0 0 0 0 1 2.04e-04 0
0.00e+00 0
SFUnpack 200 1.0 2.3165e-05 1.4 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00 0 0 0 0 0 0 0 0 0 0 0 0 0 0.00e+00 0
0.00e+00 0
> On Jan 25, 2022, at 8:29 AM, Mark Adams <[email protected]> wrote:
>
> adding Suyash,
>
> I found the/a problem. Using ex56, which has a crappy decomposition, using
> one MPI process/GPU is much faster than using 8 (64 total). (I am looking at
> ex13 to see how much of this is due to the decomposition)
> If you only use 8 processes it seems that all 8 are put on the first GPU, but
> adding -c8 seems to fix this.
> Now the numbers are looking reasonable.
>
> On Mon, Jan 24, 2022 at 3:24 PM Barry Smith <[email protected]
> <mailto:[email protected]>> wrote:
>
> For this, to start, someone can run
>
> src/vec/vec/tutorials/performance.c
>
> and compare the performance to that in the technical report Evaluation of
> PETSc on a Heterogeneous Architecture \\ the OLCF Summit System \\ Part I:
> Vector Node Performance. Google to find. One does not have to and shouldn't
> do an extensive study right now that compares everything, instead one should
> run a very small number of different size problems (make them big) and
> compare those sizes with what Summit gives. Note you will need to make sure
> that performance.c uses the Kokkos backend.
>
> One hopes for better performance than Summit; if one gets tons worse we
> know something is very wrong somewhere. I'd love to see some comparisons.
>
> Barry
>
>
>> On Jan 24, 2022, at 3:06 PM, Justin Chang <[email protected]
>> <mailto:[email protected]>> wrote:
>>
>> Also, do you guys have an OLCF liaison? That's actually your better bet if
>> you do.
>>
>> Performance issues with ROCm/Kokkos are pretty common in apps besides just
>> PETSc. We have several teams actively working on rectifying this. However, I
>> think performance issues can be quicker to identify if we had a more
>> "official" and reproducible PETSc GPU benchmark, which I've already
>> expressed to some folks in this thread, and as others already commented on
>> the difficulty of such a task. Hopefully I will have more time soon to
>> illustrate what I am thinking.
>>
>> On Mon, Jan 24, 2022 at 1:57 PM Justin Chang <[email protected]
>> <mailto:[email protected]>> wrote:
>> My name has been called.
>>
>> Mark, if you're having issues with Crusher, please contact Veronica Vergara
>> ([email protected] <mailto:[email protected]>). You can cc me
>> ([email protected] <mailto:[email protected]>) in those emails
>>
>> On Mon, Jan 24, 2022 at 1:49 PM Barry Smith <[email protected]
>> <mailto:[email protected]>> wrote:
>>
>>
>>> On Jan 24, 2022, at 2:46 PM, Mark Adams <[email protected]
>>> <mailto:[email protected]>> wrote:
>>>
>>> Yea, CG/Jacobi is as close to a benchmark code as we could want. I could
>>> run this on one processor to get cleaner numbers.
>>>
>>> Is there a designated ECP technical support contact?
>>
>> Mark, you've forgotten you work for DOE. There isn't a non-ECP technical
>> support contact.
>>
>> But if this is an AMD machine then maybe contact Matt's student Justin
>> Chang?
>>
>>
>>
>>>
>>>
>>> On Mon, Jan 24, 2022 at 2:18 PM Barry Smith <[email protected]
>>> <mailto:[email protected]>> wrote:
>>>
>>> I think you should contact the crusher ECP technical support team and
>>> tell them you are getting dismel performance and ask if you should expect
>>> better. Don't waste time flogging a dead horse.
>>>
>>>> On Jan 24, 2022, at 2:16 PM, Matthew Knepley <[email protected]
>>>> <mailto:[email protected]>> wrote:
>>>>
>>>> On Mon, Jan 24, 2022 at 2:11 PM Junchao Zhang <[email protected]
>>>> <mailto:[email protected]>> wrote:
>>>>
>>>>
>>>> On Mon, Jan 24, 2022 at 12:55 PM Mark Adams <[email protected]
>>>> <mailto:[email protected]>> wrote:
>>>>
>>>>
>>>> On Mon, Jan 24, 2022 at 1:38 PM Junchao Zhang <[email protected]
>>>> <mailto:[email protected]>> wrote:
>>>> Mark, I think you can benchmark individual vector operations, and once we
>>>> get reasonable profiling results, we can move to solvers etc.
>>>>
>>>> Can you suggest a code to run or are you suggesting making a vector
>>>> benchmark code?
>>>> Make a vector benchmark code, testing vector operations that would be used
>>>> in your solver.
>>>> Also, we can run MatMult() to see if the profiling result is reasonable.
>>>> Only once we get some solid results on basic operations, it is useful to
>>>> run big codes.
>>>>
>>>> So we have to make another throw-away code? Why not just look at the
>>>> vector ops in Mark's actual code?
>>>>
>>>> Matt
>>>>
>>>>
>>>>
>>>> --Junchao Zhang
>>>>
>>>>
>>>> On Mon, Jan 24, 2022 at 12:09 PM Mark Adams <[email protected]
>>>> <mailto:[email protected]>> wrote:
>>>>
>>>>
>>>> On Mon, Jan 24, 2022 at 12:44 PM Barry Smith <[email protected]
>>>> <mailto:[email protected]>> wrote:
>>>>
>>>> Here except for VecNorm the GPU is used effectively in that most of the
>>>> time is time is spent doing real work on the GPU
>>>>
>>>> VecNorm 402 1.0 4.4100e-01 6.1 1.69e+09 1.0 0.0e+00 0.0e+00
>>>> 4.0e+02 0 1 0 0 20 9 1 0 0 33 30230 225393 0 0.00e+00 0
>>>> 0.00e+00 100
>>>>
>>>> Even the dots are very effective, only the VecNorm flop rate over the full
>>>> time is much much lower than the vecdot. Which is somehow due to the use
>>>> of the GPU or CPU MPI in the allreduce?
>>>>
>>>> The VecNorm GPU rate is relatively high on Crusher and the CPU rate is
>>>> about the same as the other vec ops. I don't know what to make of that.
>>>>
>>>> But Crusher is clearly not crushing it.
>>>>
>>>> Junchao: Perhaps we should ask Kokkos if they have any experience with
>>>> Crusher that they can share. They could very well find some low level
>>>> magic.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> On Jan 24, 2022, at 12:14 PM, Mark Adams <[email protected]
>>>>> <mailto:[email protected]>> wrote:
>>>>>
>>>>>
>>>>>
>>>>> Mark, can we compare with Spock?
>>>>>
>>>>> Looks much better. This puts two processes/GPU because there are only 4.
>>>>> <jac_out_001_kokkos_Spock_6_1_notpl.txt>
>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>
> <jac_out_001_kokkos_Crusher_159_1.txt>
