Hi Xiang,
May I ask when do you plan to add the Hyperscan code to the DPDK?
> -----Original Message-----
> From: dev <dev-boun...@dpdk.org> On Behalf Of Wang Xiang
> Sent: Monday, March 2, 2020 9:05 AM
> To: Ori Kam <or...@mellanox.com>
> Cc: jer...@marvell.com; dev@dpdk.org; pbhagavat...@marvell.com; Shahaf
> Shuler <shah...@mellanox.com>; hemant.agra...@nxp.com; Opher Reviv
> <op...@mellanox.com>; Alex Rosenbaum <al...@mellanox.com>;
> dov...@marvell.com; pkap...@marvell.com; nipun.gu...@nxp.com;
> bruce.richard...@intel.com; yang.a.h...@intel.com; harry.ch...@intel.com;
> gu.ji...@zte.com.cn; shanjia...@chinatelecom.cn;
> zhangy....@chinatelecom.cn; lixin...@huachentel.com; wush...@inspur.com;
> yuying...@yxlink.com; fanchengg...@sunyainfo.com;
> davidf...@tencent.com; liuzho...@chinaunicom.cn;
> zhaoyon...@huawei.com; o...@yunify.com; j...@netgate.com;
> hongjun...@intel.com; j.bromh...@titan-ic.com; d...@ntop.org;
> f...@napatech.com; arthur...@lionic.com; Thomas Monjalon
> <tho...@monjalon.net>
> Subject: Re: [dpdk-dev] [RFC v5] regexdev: introduce regexdev subsystem
>
> Hi Ori,
>
> Comments below.
>
> Thanks,
> Xiang
>
> On Thu, Feb 27, 2020 at 03:08:35PM +0000, Ori Kam wrote:
> > From: Jerin Jacob <jer...@marvell.com>
> >
> > Even though there are some vendors which offer Regex HW offload, due to
> > lack of standard API, It is diffcult for DPDK consumer to use them
> > in a portable way.
> >
> > This _RFC_ attempts to standardize the RegEx/DPI offload APIs for DPDK.
> >
> > This RFC crafted based on SW Regex API frameworks such as libpcre and
> > hyperscan and a few of the RegEx HW IPs which I am aware of.
> >
> > RegEx pattern matching applications:
> > * Next Generation Firewalls (NGFW)
> > * Deep Packet and Flow Inspection (DPI)
> > * Intrusion Prevention Systems (IPS)
> > * DDoS Mitigation
> > * Network Monitoring
> > * Data Loss Prevention (DLP)
> > * Smart NICs
> > * Grammar based content processing
> > * URL, spam and adware filtering
> > * Advanced auditing and policing of user/application security policies
> > * Financial data mining - parsing of streamed financial feeds
> > * Application recognition.
> > * Dmemory introspection.
> > * Natural Language Processing (NLP)
> > * Sentiment Analysis.
> > * Big data databse acceleration.
> > * Computational storage.
> >
> > Request to review from HW and SW RegEx vendors and RegEx application
> > users to have portable DPDK API for RegEx.
> >
> > The API schematics are based cryptodev, eventdev and ethdev existing
> > device API.
> >
> > Signed-off-by: Jerin Jacob <jer...@marvell.com>
> > Signed-off-by: Pavan Nikhilesh <pbhagavat...@marvell.com>
> > Signed-off-by: Ori Kam <or...@mellanox.com>
> > ---
> > V5:
> > * Remove unused iov struct.
> > V4:
> > * Replace iov with mbuf.
> > * Small ML comments.
> > V3:
> > * Change subject title.
> > V2:
> > * Address ML comments.
> > ---
> > +
> > +#define RTE_REGEX_DEV_SUPP_PCRE_GREEDY_F (1ULL << 6)
> > +/**< RegEx device support PCRE Greedy mode.
> > + * For example if the RegEx is 'AB\d*?' then '*?' represents zero or
> unlimited
> > + * matches. In greedy mode the pattern 'AB12345' will be matched
> completely
> > + * where as the ungreedy mode 'AB' will be returned as the match.
> > + * @see struct rte_regex_dev_info::regex_dev_capa
> > + */
> > +
>
> Hyperscan actually supports "match all" semantic, neither greedy nor
> ungreedy,
> which is different from PCRE. In the case above, AB, AB1, ..., AB12345 will
> all
> be returned as matches. Do HW solutions support this?
No our HW doesn't support this.
Jerin, does Marvell HW support this?
> Can we add a new flag like RTE_REGEX_DEV_SUPP_PCRE_MATCHALL_F?
> Similarly, we can define a flag RTE_REGEX_PCRE_RULE_MATCHALL_F so
> Hyperscan
> users have to set this flag during rule compile.
>
Sure,
> > +#define RTE_REGEX_DEV_SUPP_PCRE_LOOKAROUND_ASRT_F (1ULL << 7)
> > +/**< RegEx device support PCRE Lookaround assertions
> > + * (Zero-width assertions). Example RegEx is '[a-z]+\d+(?=!{3,})' if
> > + * the given pattern is 'dwad1234!' the RegEx engine doesn't report any
> matches
> > + * because the assert '(?=!{3,})' fails. The pattern 'dwad123!!!' would
> > return
> a
> > + * successful match.
> > + * @see struct rte_regex_dev_info::regex_dev_capa
> > + */
> > +
> > +
> > +/**
> > + * RegEx device information
> > + */
> > +struct rte_regex_dev_info {
> > + const char *driver_name; /**< RegEx driver name. */
> > + struct rte_device *dev; /**< Device information. */
> > + uint16_t max_matches;
> > + /**< Maximum matches per scan supported by this device. */
> > + uint16_t max_queue_pairs;
> > + /**< Maximum queue pairs supported by this device. */
> > + uint16_t max_payload_size;
> > + /**< Maximum payload size for a pattern match request or scan.
> > + * @see RTE_REGEX_DEV_CFG_CROSS_BUFFER_SCAN_F
> > + */
> > + uint32_t max_rules_per_group;
> > + /**< Maximum rules supported per group by this device. */
> > + uint16_t max_groups;
> > + /**< Maximum groups supported by this device. */
> > + uint32_t regex_dev_capa;
> > + /**< RegEx device capabilities. @see RTE_REGEX_DEV_CAPA_* */
> > + uint64_t rule_flags;
> > + /**< Supported compiler rule flags.
> > + * @see RTE_REGEX_PCRE_RULE_*, struct rte_regex_rule::rule_flags
> > + */
> > + uint8_t max_scatter_gather;
> > + /**< The max supported number of buffers that can
> > + * be used in a single ops. The total size of all elements
> > + * must be less then max_payload_size.
> > + */
>
> s/then/than
>
Will fix.
> > +};
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Retrieve the contextual information of a RegEx device.
> > + *
> > + * @param dev_id
> > + * The identifier of the device.
> > + *
> > + * @param[out] dev_info
> > + * A pointer to a structure of type *rte_regex_dev_info* to be filled
> > with
> the
> > + * contextual information of the device.
> > + *
> > + * @return
> > + * - 0: Success, driver updates the contextual information of the RegEx
> device
> > + * - <0: Error code returned by the driver info get function.
> > + *
> > + */
> > +__rte_experimental
> > +int
> > +rte_regex_dev_info_get(uint8_t dev_id, struct rte_regex_dev_info
> *dev_info);
> > +
> > +/* Enumerates RegEx device configuration flags */
> > +#define RTE_REGEX_DEV_CFG_CROSS_BUFFER_SCAN_F (1ULL << 0)
> > +/**< Cross buffer scan refers to the ability to be able to detect
> > + * matches that occur across buffer boundaries, where the buffers are
> related
> > + * to each other in some way. Enable this flag when to scan payload size
> > + * greater struct struct rte_regex_dev_info::max_payload_size and/or
> > + * matches can present across scan buffer boundaries.
>
> s/struct/than
>
Will fix.
> > + *
> > + * @see struct rte_regex_dev_info::max_payload_size
> > + * @see struct rte_regex_dev_config::dev_cfg_flags,
> rte_regex_dev_configure()
> > + * @see RTE_REGEX_OPS_RSP_PMI_SOJ_F
> > + * @see RTE_REGEX_OPS_RSP_PMI_EOJ_F
> > + */
> > +
> > +
> > +/* Enumerates RegEx response flags. */
> > +#define RTE_REGEX_OPS_RSP_PMI_SOJ_F (1 << 0)
> > +/**< Indicates that the RegEx device has encountered a partial match at the
> > + * start of scan in the given buffer.
> > + *
> > + * @see RTE_REGEX_DEV_CFG_CROSS_BUFFER_SCAN_F
> > + */
> > +
> Hyperscan supports cross buffer scan and only reports true matches instead of
> partial matches. Can we have users to config this partial match capability?
>
Do you mean something like this:
RTE_REGEX_OPS_RSP_PMI_FOJ_F
/**< Indicates that the RegEx device has encountered a full match in the
current buffer.
* The match was started in previous buffer.
*/
> > +#define RTE_REGEX_OPS_RSP_PMI_EOJ_F (1 << 1)
> > +/**< Indicates that the RegEx device has encountered a partial match at the
> > + * end of scan in the given buffer.
> > + *
> > + * @see RTE_REGEX_DEV_CFG_CROSS_BUFFER_SCAN_F
> > + */
> > +
