Barry Smith <[email protected]> writes:
> Currently we use PetscScalar and PetscScalar * to refer to variables that
> could be in any memory space. On the CPU, on the GPU, in Kokkos, etc.
>
> Would it make sense to use typedef to indicate at each location the true
> type of the memory location when possible?
>
> typedef PetscReal PetscKokkosReal means the variable is in the Kokkos
> memory space
> typedef PetscReal PetscCUDAReal
> typedef PetscReal PetscNVSHEMReal
There's no safety here. I suppose you could do a struct to prevent implicit
conversion
typdef struct {PetscReal val;} PetscCUDAReal;
but even this doesn't enforce the invariant.
I think the distinction only makes sense for boxed values/arrays. If you ever
have a value, it's in your memory space, but it matters when you dereference a
pointer.
A question is how you'd convert. Presumably it requires a cudaMemcpy, so you'd
want the array length to be part of the type and some light interfaces to do
the conversions.
Some mapping of values to device memory can be piggy-backed on kernel launches
so I'm not sure that requiring up-front conversion is the right approach
(effectively adding one or more cudaMemcpy to the critical path).
> etc.
>
> Then one could write things like
>
> struct {
> ...
> PetscNVSHEMReal *values;
> }
>
> Similarly inside kernels one would use the type type associated with the
> kernel, cuda with cuda etc.
>
> I think the resulting code will be much clearer and easier to dive into,
> then having to first figure out where each variable lives.
>
> I find the current code confusing because one cannot immediately see a
> variable declaration and know where it lives, even though it does live
> somewhere in particular..
>
> Barry
