On 25-Sep-18 3:08 PM, Burakov, Anatoly wrote:
On 25-Sep-18 3:00 PM, Shreyansh Jain wrote:
On Tuesday 25 September 2018 07:21 PM, Burakov, Anatoly wrote:
On 25-Sep-18 2:39 PM, Shreyansh Jain wrote:
Hello Anatoly,
On Tuesday 25 September 2018 06:58 PM, Burakov, Anatoly wrote:
On 25-Sep-18 1:54 PM, Shreyansh Jain wrote:
A common library, valid for dpaaX drivers, which is used to maintain
a local copy of PA->VA translations.
In case of physical addressing mode (one of the option for FSLMC, and
only option for DPAA bus), the addresses of descriptors Rx'd are
physical. These need to be converted into equivalent VA for rte_mbuf
and other similar calls.
Using the rte_mem_virt2iova or rte_mem_virt2phy is expensive. This
library is an attempt to reduce the overall cost associated with
this translation.
A small table is maintained, containing continuous entries
representing a continguous physical range. Each of these entries
stores the equivalent VA, which is fed during mempool creation, or
memory allocation/deallocation callbacks.
Signed-off-by: Shreyansh Jain <shreyansh.j...@nxp.com>
---
Hi Shreyansh,
So, basically, you're reimplementing old DPDK's memory view
(storing VA's in a PA-centric way). Makes sense :)
Yes, and frankly, I couldn't come up with any other way.
I should caution you that right now, external memory allocator
implementation does *not* trigger any callbacks for newly added
memory. So, anything coming from external memory will not be
reflected in your table, unless it happens to be already there
before dpaax_iova_table_populate() gets called. This patchset makes
a good argument for why perhaps it should trigger callbacks. Thoughts?
Oh. Then I must be finishing reading through your patches for
external memory sooner. I didn't realize this.
To be clear, the current implementation of external memory allocators
is not necessarily final - it's not too late to add callbacks to
enable your use case better, if that's required (and it should be
pretty easy to implement as well).
Is there any reason why external may not be raising call back right
now? I might have missed any previous conversation on this. Or may be,
it is just lack of need.
Well, pretty much - it didn't occur to me that it may be needed. I
specifically went out of my way to note that it is the responsibility of
the user to perform any DMA mappings, but i missed the fact that there
may be other users interested to know that a user has just added a new
external memory segment.
As for whether it is required - I do see a need. It is definitely
possible that after rte_eal_init has been completed (and underlying
probe), applications allocate memory. In which case, even existing
memevent callbacks (like the one in fslmc_bus, which VFIO/DMA maps the
area) would have issues. From the external memory patchset, I do see
that it is assumed DMA mapping is caller's responsibility.
Having such callback would help drives reduce that throwback of
responsibility.
I do not want to assume that user necessarily wants to map external
memory for DMA unless explicitly asked to do so. At the same time, i can
see that some uses may not have anything to do with DMA mapping and may
instead be cases like yours, where you just need the address. In our
case, we can just ignore external memory in VFIO and virtio callbacks,
but still allow other callbacks to handle external memory as they see fit.
(Speaking of external memory patches, I also realize that my memevent
callback in this patch series need to handle msl->external).
Yes, we have to be careful on merge.
Hi Shreyansh,
I'm currently implementing callback support for external memory. I think
the decision to leave DMA mapping to the user may have been a bad one.
For starters, one of the buses (bus/fslmc) has its own VFIO
infrastructure independent of EAL's VFIO, so if we don't map it there -
we can't map it at all because there's no generic manual way to do a DMA
map with bus/fslmc driver after the fact.
Also, if we leave VFIO mapping to the user, we end up with an
inconsistency where we provide memory callbacks which can potentially
create DMA mappings (such as what some MLX drivers do), but VFIO DMA
mappings will be ignored because "reasons".
The only place where we really *don't* want to see external memory is
virtio, because there is no segment fd support for external memory yet.
Every other place, i think it's a good idea to not skip external memory.
So, starting from v5, DMA mapping will be performed automatically for
external memory when IOVA addresses are available.
--
Thanks,
Anatoly
