Hi Anatoly,
Marvell CNXK DMA hardware also supports this feature, and it would be a good
feature to add. Thanks for introducing the feature. Please see inline.
Thanks,
Anoob
> -----Original Message-----
> From: Anatoly Burakov <anatoly.bura...@intel.com>
> Sent: Friday, August 11, 2023 9:45 PM
> To: dev@dpdk.org; Chengwen Feng <fengcheng...@huawei.com>; Kevin
> Laatz <kevin.la...@intel.com>; Bruce Richardson
> <bruce.richard...@intel.com>
> Cc: Vladimir Medvedkin <vladimir.medved...@intel.com>
> Subject: [EXT] [PATCH v1 1/3] dmadev: add inter-domain operations
>
> External Email
>
> ----------------------------------------------------------------------
> Add a flag to indicate that a specific device supports inter-domain
> operations,
> and add an API for inter-domain copy and fill.
>
> Inter-domain operation is an operation that is very similar to regular DMA
> operation, except either source or destination addresses can be in a
> different process's address space, indicated by source and destination
> handle values. These values are currently meant to be provided by private
> drivers' API's.
>
> This commit also adds a controller ID field into the DMA device API.
> This is an arbitrary value that may not be implemented by hardware, but it is
> meant to represent some kind of device hierarchy.
>
> Signed-off-by: Vladimir Medvedkin <vladimir.medved...@intel.com>
> Signed-off-by: Anatoly Burakov <anatoly.bura...@intel.com>
> ---
> doc/guides/prog_guide/dmadev.rst | 18 +++++
> lib/dmadev/rte_dmadev.c | 2 +
> lib/dmadev/rte_dmadev.h | 133
> +++++++++++++++++++++++++++++++
> lib/dmadev/rte_dmadev_core.h | 12 +++
> 4 files changed, 165 insertions(+)
>
<snip>
> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change without prior notice.
> + *
> + * Enqueue an inter-domain copy operation.
> + *
> + * This queues up an inter-domain copy operation to be performed by
> +hardware, if
> + * the 'flags' parameter contains RTE_DMA_OP_FLAG_SUBMIT then trigger
> +doorbell
> + * to begin this operation, otherwise do not trigger doorbell.
> + *
> + * The source and destination handle parameters are arbitrary opaque
> +values,
> + * currently meant to be provided by private device driver API's. If
> +the source
> + * handle value is meaningful, RTE_DMA_OP_FLAG_SRC_HANDLE flag must
> be set.
> + * Similarly, if the destination handle value is meaningful,
> + * RTE_DMA_OP_FLAG_DST_HANDLE flag must be set. Source and
> destination
> +handle
> + * values are meant to provide information to the hardware about source
> +and/or
> + * destination PASID for the inter-domain copy operation.
> + *
> + * @param dev_id
> + * The identifier of the device.
> + * @param vchan
> + * The identifier of virtual DMA channel.
> + * @param src
> + * The address of the source buffer (if `src_handle` is set, source address
> + * will be in address space of process referred to by source handle).
> + * @param dst
> + * The address of the destination buffer (if `dst_handle` is set,
> destination
> + * address will be in address space of process referred to by destination
> + * handle).
> + * @param length
> + * The length of the data to be copied.
> + * @param src_handle
> + * Source handle value (if used, RTE_DMA_OP_FLAG_SRC_HANDLE flag
> must be set).
> + * @param dst_handle
> + * Destination handle value (if used, RTE_DMA_OP_FLAG_DST_HANDLE
> flag must be
> + * set).
> + * @param flags
> + * Flags for this operation.
> + * @return
> + * - 0..UINT16_MAX: index of enqueued job.
> + * - -ENOSPC: if no space left to enqueue.
> + * - other values < 0 on failure.
> + */
> +__rte_experimental
> +static inline int
> +rte_dma_copy_inter_dom(int16_t dev_id, uint16_t vchan, rte_iova_t src,
> + rte_iova_t dst, uint32_t length, uint16_t src_handle,
> + uint16_t dst_handle, uint64_t flags)
> +{
[Anoob] Won't this lead to duplication of all datapath APIs? Also, this
approach assumes that 'inter-domain' operations always support run-time setting
of 'src_handle' and 'dst_handle' within one DMA channel, which need not be
supported by all platforms.
Can we move this 'src_handle' and 'dst_handle' registration to
rte_dma_vchan_setup so that the 'src_handle' and 'dst_handle' can be configured
in control path and the existing datapath APIs can work as is. The op flags
(that is proposed) can be used to determine whether 'inter-domain' operation is
requested. Having a fixed 'src_handle' & 'dst_handle' per vchan would be better
for performance as well.
<snip>
