Re: [dpdk-dev] [dpdk-stable] [PATCH v2 07/13] app/testpmd: fix lack of flow action configuration

2018-04-04 Thread Nélio Laranjeiro
ect. */ > ctx->object = out->args.vc.data; > ctx->objmask = NULL; > - /* Update configuration pointer. */ > - action->conf = ctx->object; > return len; > } > > -- > 2.11.0 -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [dpdk-stable] [PATCH v2 08/13] app/testpmd: fix RSS flow action configuration

2018-04-05 Thread Nélio Laranjeiro
uot;) > Cc: sta...@dpdk.org > > Signed-off-by: Adrien Mazarguil > Cc: Wenzhuo Lu > Cc: Jingjing Wu Acked-by: Nelio Laranjeiro -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [dpdk-stable] [PATCH v2 09/13] app/testpmd: fix missing RSS fields in flow action

2018-04-05 Thread Nélio Laranjeiro
05d34c6e9d2c ("app/testpmd: add queue actions to flow command") > Cc: sta...@dpdk.org > > Signed-off-by: Adrien Mazarguil > Cc: Wenzhuo Lu > Cc: Jingjing Wu Acked-by: Nelio Laranjeiro -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [dpdk-stable] [PATCH v2 11/13] ethdev: fix missing boolean values in flow command

2018-04-05 Thread Nélio Laranjeiro
"no", "yes", > "N", "Y", > + "off", "on", > NULL, > }; > > -- > 2.11.0 -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [dpdk-stable] [PATCH v2 13/13] ethdev: fix missing include in flow API

2018-04-05 Thread Nélio Laranjeiro
flow rules. > */ > > +#include > + > #include > #include > #include > -- > 2.11.0 -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v1 11/16] ethdev: refine TPID handling in flow API

2018-04-05 Thread Nélio Laranjeiro
mask->inner_type != RTE_BE16(0x)) > + return -1; > + info->eth_type = spec->inner_type; > + } else { > + info->eth_type = 0; > + } > if (!flow) > return 0; > msg = &flow->msg; > diff --git a/lib/librte_ether/rte_flow.h b/lib/librte_ether/rte_flow.h > index 1b222ba60..15e383f95 100644 > --- a/lib/librte_ether/rte_flow.h > +++ b/lib/librte_ether/rte_flow.h > @@ -454,11 +454,17 @@ static const struct rte_flow_item_raw > rte_flow_item_raw_mask = { > * RTE_FLOW_ITEM_TYPE_ETH > * > * Matches an Ethernet header. > + * > + * The @p type field either stands for "EtherType" or "TPID" when followed > + * by so-called layer 2.5 pattern items such as RTE_FLOW_ITEM_TYPE_VLAN. In > + * the latter case, @p type refers to that of the outer header, with the > + * inner EtherType/TPID provided by the subsequent pattern item. This is the > + * same order as on the wire. > */ > struct rte_flow_item_eth { > struct ether_addr dst; /**< Destination MAC. */ > struct ether_addr src; /**< Source MAC. */ > - rte_be16_t type; /**< EtherType. */ > + rte_be16_t type; /**< EtherType or TPID. */ > }; > > /** Default mask for RTE_FLOW_ITEM_TYPE_ETH. */ > @@ -475,19 +481,20 @@ static const struct rte_flow_item_eth > rte_flow_item_eth_mask = { > * > * Matches an 802.1Q/ad VLAN tag. > * > - * This type normally follows either RTE_FLOW_ITEM_TYPE_ETH or > - * RTE_FLOW_ITEM_TYPE_VLAN. > + * The corresponding standard outer EtherType (TPID) values are 0x8100 or > + * 0x88a8 (in case of outer QinQ). It can be overridden by the preceding > + * pattern item. > */ > struct rte_flow_item_vlan { > - rte_be16_t tpid; /**< Tag protocol identifier. */ > rte_be16_t tci; /**< Tag control information. */ > + rte_be16_t inner_type; /**< Inner EtherType or TPID. */ > }; > > /** Default mask for RTE_FLOW_ITEM_TYPE_VLAN. */ > #ifndef __cplusplus > static const struct rte_flow_item_vlan rte_flow_item_vlan_mask = { > - .tpid = RTE_BE16(0x), > - .tci = RTE_BE16(0x), > + .tci = RTE_BE16(0x0fff), > + .inner_type = RTE_BE16(0x), > }; > #endif > > @@ -636,9 +643,11 @@ static const struct rte_flow_item_vxlan > rte_flow_item_vxlan_mask = { > * RTE_FLOW_ITEM_TYPE_E_TAG. > * > * Matches a E-tag header. > + * > + * The corresponding standard outer EtherType (TPID) value is 0x893f. > + * It can be overridden by the preceding pattern item. > */ > struct rte_flow_item_e_tag { > - rte_be16_t tpid; /**< Tag protocol identifier (0x893F). */ > /** >* E-Tag control information (E-TCI). >* E-PCP (3b), E-DEI (1b), ingress E-CID base (12b). > @@ -648,6 +657,7 @@ struct rte_flow_item_e_tag { > rte_be16_t rsvd_grp_ecid_b; > uint8_t in_ecid_e; /**< Ingress E-CID ext. */ > uint8_t ecid_e; /**< E-CID ext. */ > + rte_be16_t inner_type; /**< Inner EtherType or TPID. */ > }; > > /** Default mask for RTE_FLOW_ITEM_TYPE_E_TAG. */ > -- > 2.11.0 -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH] net/mlx5: fix link status initialization

2018-04-09 Thread Nélio Laranjeiro
On Sun, Apr 08, 2018 at 01:09:27PM +, Shahaf Shuler wrote: > Thursday, April 5, 2018 9:51 AM, Nélio Laranjeiro: > > Subject: Re: [PATCH] net/mlx5: fix link status initialization > > > > On Thu, Apr 05, 2018 at 05:35:57AM +, Shahaf Shuler wrote: > > > We

Re: [dpdk-dev] [PATCH] ethdev: return diagnostic when setting MAC address

2018-04-09 Thread Nélio Laranjeiro
rt %u cannot set mac address: %s", > dev->data->port_id, strerror(rte_errno)); > + return ret; > } You should also remove the DRV_LOG which become useless, it has been added as the function did not return anything, now has it returns an error, the application can fully handle it. Unless that, for MLX5 Acked-by: Nelio Laranjeiro -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH] net/mlx5: fix link status initialization

2018-04-09 Thread Nélio Laranjeiro
On Mon, Apr 09, 2018 at 12:28:04PM +, Shahaf Shuler wrote: > Monday, April 9, 2018 11:28 AM, Nélio Laranjeiro: > > Subject: Re: [PATCH] net/mlx5: fix link status initialization > > > > On Sun, Apr 08, 2018 at 01:09:27PM +, Shahaf Shuler wrote: > > > Thursday

Re: [dpdk-dev] [PATCH] net/mlx5: fix link status initialization

2018-04-09 Thread Nélio Laranjeiro
On Mon, Apr 09, 2018 at 03:26:41PM +0200, Nélio Laranjeiro wrote: > On Mon, Apr 09, 2018 at 12:28:04PM +, Shahaf Shuler wrote: > > Monday, April 9, 2018 11:28 AM, Nélio Laranjeiro: > > > Subject: Re: [PATCH] net/mlx5: fix link status initialization > > > > >

Re: [dpdk-dev] [PATCH v2] net/mlx5: fix link status initialization

2018-04-10 Thread Nélio Laranjeiro
priv->config = config; > continue; > -- > 2.12.0 You should have linked this patch with the ethdev one[1], if the ethdev is refused, this patch won't solve anything. With a reserves of acceptance of [1]: Acked-by: Nelio Laranjeiro Thanks, [1] https://dpdk.org/ml/archives/dev/2018-April/096387.html -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH] ethdev: fix link status query

2018-04-10 Thread Nélio Laranjeiro
RTE_ETH_VALID_PORTID_OR_RET(port_id); > dev = &rte_eth_devices[port_id]; > > - if (dev->data->dev_conf.intr_conf.lsc) > + if (dev->data->dev_conf.intr_conf.lsc && > + dev->data->dev_started) > rte_eth_linkstatus_get(

Re: [dpdk-dev] [PATCH v2 4/5] app/testpmd: introduce new tunnel VXLAN-GPE

2018-04-10 Thread Nélio Laranjeiro
rt must be >= > 0\n"); > + } > if (!strcmp(lgopts[opt_idx].name, "print-event")) > if (parse_event_printing_config(optarg, 1)) { > rte_exit(EXIT_FAILURE, > diff --git a/app/test-pmd/testpmd.h b/app/test-pmd/testpmd.h > index 593ae2160..b270602d9 100644 > --- a/app/test-pmd/testpmd.h > +++ b/app/test-pmd/testpmd.h > @@ -446,6 +446,8 @@ extern uint32_t retry_enabled; > extern struct fwd_lcore **fwd_lcores; > extern struct fwd_stream **fwd_streams; > > +extern uint16_t vxlan_gpe_udp_port; /**< UDP port of tunnel VXLAN-GPE. */ > + > extern portid_t nb_peer_eth_addrs; /**< Number of peer ethernet addresses. */ > extern struct ether_addr peer_eth_addrs[RTE_MAX_ETHPORTS]; > > diff --git a/doc/guides/testpmd_app_ug/run_app.rst > b/doc/guides/testpmd_app_ug/run_app.rst > index 1fd53958a..2e8690f41 100644 > --- a/doc/guides/testpmd_app_ug/run_app.rst > +++ b/doc/guides/testpmd_app_ug/run_app.rst > @@ -479,3 +479,8 @@ The commandline options are: > > Set the hexadecimal bitmask of TX queue offloads. > The default value is 0. > + > +* ``--vxlan-gpe-port=N`` > + > +Set the UDP port number of tunnel VXLAN-GPE to N. > +The default value is 4790. > -- > 2.13.3 > -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v2 01/15] net/mlx5: support 16 hardware priorities

2018-04-10 Thread Nélio Laranjeiro
, "port %u allocating and configuring hash Rx queues", > dev->data->port_id); > rte_mempool_walk(mlx5_mp2mr_iter, priv); > @@ -202,7 +196,6 @@ mlx5_dev_start(struct rte_eth_dev *dev) > mlx5_traffic_disable(dev); > mlx5_txq_stop(dev); > mlx5_rxq_stop(dev); > - mlx5_flow_delete_drop_queue(dev); > rte_errno = ret; /* Restore rte_errno. */ > return -rte_errno; > } > @@ -237,7 +230,6 @@ mlx5_dev_stop(struct rte_eth_dev *dev) > mlx5_rxq_stop(dev); > for (mr = LIST_FIRST(&priv->mr); mr; mr = LIST_FIRST(&priv->mr)) > mlx5_mr_release(mr); > - mlx5_flow_delete_drop_queue(dev); > } > > /** > -- > 2.13.3 I have few concerns on this, mlx5_pci_probe() will also probe any under layer verbs device, and in a near future the representors associated to a VF. Making such detection should only be done once by the PF, I also wander if it is possible to make such drop action in a representor directly using Verbs. Another concern is, this patch will be reverted in some time when those 16 priority will be always available. It will be easier to remove this detection function than searching for all those modifications. I would suggest to have a standalone mlx5_flow_priorities_detect() which creates and deletes all resources needed for this detection. Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v2 03/15] net/mlx5: support L3 vxlan flow

2018-04-10 Thread Nélio Laranjeiro
VXLAN. It also deserves a comment in the code itself, currently this seems a bug as the RFC [1] implies to have an inner Ethernet layer. I suppose there is a use case for such modification but as it not explained it is just a supposition. Thanks, [1] https://tools.ietf.org/html/rfc7348 -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v2 04/15] net/mlx5: support Rx tunnel type identification

2018-04-10 Thread Nélio Laranjeiro
!(pt_idx3 & (1 << 6)) * rxq->tunnel; > /* Fill flags for checksum and VLAN. */ > pinfo = vandq_u32(ptype_info, ptype_ol_mask); > pinfo = vreinterpretq_u32_u8( > diff --git a/drivers/net/mlx5/mlx5_rxtx_vec_sse.h > b/drivers/net/mlx5/mlx5_rxtx_vec_sse.h > index c088bcb51..d2492481d 100644 > --- a/drivers/net/mlx5/mlx5_rxtx_vec_sse.h > +++ b/drivers/net/mlx5/mlx5_rxtx_vec_sse.h > @@ -542,6 +542,7 @@ rxq_cq_to_ptype_oflags_v(struct mlx5_rxq_data *rxq, > __m128i cqes[4], > const __m128i mbuf_init = > _mm_loadl_epi64((__m128i *)&rxq->mbuf_initializer); > __m128i rearm0, rearm1, rearm2, rearm3; > + uint8_t pt_idx0, pt_idx1, pt_idx2, pt_idx3; > > /* Extract pkt_info field. */ > pinfo0 = _mm_unpacklo_epi32(cqes[0], cqes[1]); > @@ -595,10 +596,18 @@ rxq_cq_to_ptype_oflags_v(struct mlx5_rxq_data *rxq, > __m128i cqes[4], > /* Errored packets will have RTE_PTYPE_ALL_MASK. */ > op_err = _mm_srli_epi16(op_err, 8); > ptype = _mm_or_si128(ptype, op_err); > - pkts[0]->packet_type = mlx5_ptype_table[_mm_extract_epi8(ptype, 0)]; > - pkts[1]->packet_type = mlx5_ptype_table[_mm_extract_epi8(ptype, 2)]; > - pkts[2]->packet_type = mlx5_ptype_table[_mm_extract_epi8(ptype, 4)]; > - pkts[3]->packet_type = mlx5_ptype_table[_mm_extract_epi8(ptype, 6)]; > + pt_idx0 = _mm_extract_epi8(ptype, 0); > + pt_idx1 = _mm_extract_epi8(ptype, 2); > + pt_idx2 = _mm_extract_epi8(ptype, 4); > + pt_idx3 = _mm_extract_epi8(ptype, 6); > + pkts[0]->packet_type = mlx5_ptype_table[pt_idx0] | > +!!(pt_idx0 & (1 << 6)) * rxq->tunnel; > + pkts[1]->packet_type = mlx5_ptype_table[pt_idx1] | > +!!(pt_idx1 & (1 << 6)) * rxq->tunnel; > + pkts[2]->packet_type = mlx5_ptype_table[pt_idx2] | > +!!(pt_idx2 & (1 << 6)) * rxq->tunnel; > + pkts[3]->packet_type = mlx5_ptype_table[pt_idx3] | > +!!(pt_idx3 & (1 << 6)) * rxq->tunnel; > /* Fill flags for checksum and VLAN. */ > pinfo = _mm_and_si128(pinfo, ptype_ol_mask); > pinfo = _mm_shuffle_epi8(cv_flag_sel, pinfo); > -- > 2.13.3 Regards, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v2 05/15] net/mlx5: support tunnel inner checksum offloads

2018-04-10 Thread Nélio Laranjeiro
index 6866f6818..d35605b55 100644 > --- a/drivers/net/mlx5/mlx5_rxtx.h > +++ b/drivers/net/mlx5/mlx5_rxtx.h > @@ -77,7 +77,6 @@ struct rxq_zip { > /* RX queue descriptor. */ > struct mlx5_rxq_data { > unsigned int csum:1; /* Enable checksum offloading. */ > - unsigned int csum_l2tun:1; /* Same for L2 tunnels. */ > unsigned int hw_timestamp:1; /* Enable HW timestamp. */ > unsigned int vlan_strip:1; /* Enable VLAN stripping. */ > unsigned int crc_present:1; /* CRC must be subtracted. */ > -- > 2.13.3 This last part seems to introduce a regression by removing the support for the tunnel checksum offload. Seems this patch is incomplete or wrongly explained. -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v2 06/15] net/mlx5: split flow RSS handling logic

2018-04-10 Thread Nélio Laranjeiro
rser->rss_conf.queue_num = 1; > + } > + return 0; > +} > + > +/** > * Validate and convert a flow supported by the NIC. > * > * @param dev > @@ -1214,6 +1223,9 @@ mlx5_flow_convert(struct rte_eth_dev *dev, >* configuration. >*/ > if (!parser->drop) > + ret = mlx5_flow_convert_rss(parser); > + if (ret) > + goto exit_free; > mlx5_flow_convert_finalise(parser); > mlx5_flow_update_priority(dev, parser, attr); > exit_free: > -- > 2.13.3 > -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v2 07/15] net/mlx5: support tunnel RSS level

2018-04-11 Thread Nélio Laranjeiro
T_INSERT_HEAD(&priv->hrxqs, hrxq, next); > @@ -1448,6 +1483,8 @@ mlx5_hrxq_new(struct rte_eth_dev *dev, > return hrxq; > error: > err = rte_errno; /* Save rte_errno before cleanup. */ > + DRV_LOG(ERR, "port %u: Error creating Hash Rx queue", > + dev->data->port_id); Developer log, please remove it, for the user the flow won't be created with the correct error reported. > mlx5_ind_table_ibv_release(dev, ind_tbl); > if (qp) > claim_zero(mlx5_glue->destroy_qp(qp)); > @@ -1469,6 +1506,8 @@ mlx5_hrxq_new(struct rte_eth_dev *dev, > * Number of queues. > * @param tunnel > * Tunnel type. > + * @param rss_level > + * RSS hash on tunnel level > * > * @return > * An hash Rx queue on success. > @@ -1477,7 +1516,8 @@ struct mlx5_hrxq * > mlx5_hrxq_get(struct rte_eth_dev *dev, > const uint8_t *rss_key, uint32_t rss_key_len, > uint64_t hash_fields, > - const uint16_t *queues, uint32_t queues_n, uint32_t tunnel) > + const uint16_t *queues, uint32_t queues_n, > + uint32_t tunnel, uint32_t rss_level) Dito. > { > struct priv *priv = dev->data->dev_private; > struct mlx5_hrxq *hrxq; > @@ -1494,6 +1534,8 @@ mlx5_hrxq_get(struct rte_eth_dev *dev, > continue; > if (hrxq->tunnel != tunnel) > continue; > + if (hrxq->rss_level != rss_level) > + continue; > ind_tbl = mlx5_ind_table_ibv_get(dev, queues, queues_n); > if (!ind_tbl) > continue; > diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h > index d35605b55..62cf55109 100644 > --- a/drivers/net/mlx5/mlx5_rxtx.h > +++ b/drivers/net/mlx5/mlx5_rxtx.h > @@ -147,6 +147,7 @@ struct mlx5_hrxq { > struct ibv_qp *qp; /* Verbs queue pair. */ > uint64_t hash_fields; /* Verbs Hash fields. */ > uint32_t tunnel; /* Tunnel type. */ > + uint32_t rss_level; /* RSS on tunnel level. */ > uint32_t rss_key_len; /* Hash key length in bytes. */ > uint8_t rss_key[]; /* Hash key. */ > }; > @@ -251,12 +252,12 @@ struct mlx5_hrxq *mlx5_hrxq_new(struct rte_eth_dev *dev, > const uint8_t *rss_key, uint32_t rss_key_len, > uint64_t hash_fields, > const uint16_t *queues, uint32_t queues_n, > - uint32_t tunnel); > + uint32_t tunnel, uint32_t rss_level); > struct mlx5_hrxq *mlx5_hrxq_get(struct rte_eth_dev *dev, > const uint8_t *rss_key, uint32_t rss_key_len, > uint64_t hash_fields, > const uint16_t *queues, uint32_t queues_n, > - uint32_t tunnel); > + uint32_t tunnel, uint32_t rss_level); > int mlx5_hrxq_release(struct rte_eth_dev *dev, struct mlx5_hrxq *hxrq); > int mlx5_hrxq_ibv_verify(struct rte_eth_dev *dev); > uint64_t mlx5_get_rx_port_offloads(void); > -- > 2.13.3 > Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v2 10/15] net/mlx5: allow flow tunnel ID 0 with outer pattern

2018-04-11 Thread Nélio Laranjeiro
er proper outer spec. */ > + if (parser->out_layer == HASH_RXQ_ETH && !vxlan.val.tunnel_id) > return rte_flow_error_set(data->error, EINVAL, > RTE_FLOW_ERROR_TYPE_ITEM, > item, > -- > 2.13.3 > -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v2 12/15] doc: update mlx5 guide on tunnel offloading

2018-04-11 Thread Nélio Laranjeiro
s, [1] https://dpdk.org/doc/guides/nics/overview.html [2] https://dpdk.org/browse/dpdk/tree/doc/guides/nics/features/default.ini [3] https://dpdk.org/browse/dpdk/tree/doc/guides/nics/features/mlx5.ini -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v2 13/15] net/mlx5: setup RSS flow regardless of queue count

2018-04-11 Thread Nélio Laranjeiro
> + flow->rss_conf.level ? (uint32_t)IBV_RX_HASH_INNER : 0), > + flow->rss_conf.queue_num, > flow->frxq[i].ibv_attr->num_of_specs, > flow->frxq[i].ibv_attr->size, > flow->frxq[i].ibv_attr->priority, > -- > 2.13.3 Seems, this code should make part of "[PATCH v2 07/15] net/mlx5: support tunnel RSS level", as it re-works the code added there, in addition this feature is already present in tree for non tunnel packets. Any reason why it is not merged in the previous commit? Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v2 01/15] net/mlx5: support 16 hardware priorities

2018-04-12 Thread Nélio Laranjeiro
On Tue, Apr 10, 2018 at 03:22:46PM +, Xueming(Steven) Li wrote: > Hi Nelio, > > > -Original Message- > > From: Nélio Laranjeiro > > Sent: Tuesday, April 10, 2018 10:42 PM > > To: Xueming(Steven) Li > > Cc: Shahaf Shuler ; dev@dpdk.org > &

Re: [dpdk-dev] [PATCH v2 04/15] net/mlx5: support Rx tunnel type identification

2018-04-12 Thread Nélio Laranjeiro
On Wed, Apr 11, 2018 at 08:11:50AM +, Xueming(Steven) Li wrote: > Hi Nelio, > > > -Original Message- > > From: Nélio Laranjeiro > > Sent: Tuesday, April 10, 2018 11:17 PM > > To: Xueming(Steven) Li > > Cc: Shahaf Shuler ; dev@dpdk.org > &

Re: [dpdk-dev] [PATCH v2 01/15] net/mlx5: support 16 hardware priorities

2018-04-12 Thread Nélio Laranjeiro
On Thu, Apr 12, 2018 at 01:43:04PM +, Xueming(Steven) Li wrote: > > > > -Original Message- > > From: Nélio Laranjeiro > > Sent: Thursday, April 12, 2018 5:09 PM > > To: Xueming(Steven) Li > > Cc: Shahaf Shuler ; dev@dpdk.org > > Subject:

Re: [dpdk-dev] [PATCH v2 04/15] net/mlx5: support Rx tunnel type identification

2018-04-13 Thread Nélio Laranjeiro
On Thu, Apr 12, 2018 at 02:27:45PM +, Xueming(Steven) Li wrote: > > -Original Message- > > From: Nélio Laranjeiro > > Sent: Thursday, April 12, 2018 5:51 PM > > To: Xueming(Steven) Li > > Cc: Shahaf Shuler ; dev@dpdk.org > > Subject: Re: [PATCH v

Re: [dpdk-dev] [PATCH v3 01/14] net/mlx5: support 16 hardware priorities

2018-04-13 Thread Nélio Laranjeiro
igger.c > +++ b/drivers/net/mlx5/mlx5_trigger.c > @@ -148,12 +148,6 @@ mlx5_dev_start(struct rte_eth_dev *dev) > int ret; > > dev->data->dev_started = 1; > - ret = mlx5_flow_create_drop_queue(dev); > - if (ret) { > - DRV_LOG(ERR, "port %u drop queue allocation failed: %s", > - dev->data->port_id, strerror(rte_errno)); > - goto error; > - } > DRV_LOG(DEBUG, "port %u allocating and configuring hash Rx queues", > dev->data->port_id); > rte_mempool_walk(mlx5_mp2mr_iter, priv); > @@ -202,7 +196,6 @@ mlx5_dev_start(struct rte_eth_dev *dev) > mlx5_traffic_disable(dev); > mlx5_txq_stop(dev); > mlx5_rxq_stop(dev); > - mlx5_flow_delete_drop_queue(dev); > rte_errno = ret; /* Restore rte_errno. */ > return -rte_errno; > } > @@ -237,7 +230,6 @@ mlx5_dev_stop(struct rte_eth_dev *dev) > mlx5_rxq_stop(dev); > for (mr = LIST_FIRST(&priv->mr); mr; mr = LIST_FIRST(&priv->mr)) > mlx5_mr_release(mr); > - mlx5_flow_delete_drop_queue(dev); > } > > /** > -- > 2.13.3 Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v3 02/14] net/mlx5: support GRE tunnel flow

2018-04-13 Thread Nélio Laranjeiro
uct rte_flow_item *item __rte_unused, > + const void *default_mask __rte_unused, > + struct mlx5_flow_data *data) > +{ > + struct mlx5_flow_parse *parser = data->parser; > + unsigned int size = sizeof(struct ibv_flow_spec_tunnel); > + struct ibv_flow_spec_tunnel tunnel = { > + .type = parser->inner | IBV_FLOW_SPEC_VXLAN_TUNNEL, > + .size = size, > + }; > + > + parser->inner = IBV_FLOW_SPEC_INNER; > + mlx5_flow_create_copy(parser, &tunnel, size); > + return 0; > +} > + > +/** > * Convert mark/flag action to Verbs specification. > * > * @param parser > -- > 2.13.3 Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v3 03/14] net/mlx5: support L3 VXLAN flow

2018-04-13 Thread Nélio Laranjeiro
y such information? If both answers are no, such features should be enabled through a device parameter to let the PMD refuse such un-supported flow request. Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v3 04/14] net/mlx5: support Rx tunnel type identification

2018-04-13 Thread Nélio Laranjeiro
!(pt_idx2 & (1 << 6)) * rxq->tunnel; > + pkts[3]->packet_type = mlx5_ptype_table[pt_idx3] | > +!!(pt_idx3 & (1 << 6)) * rxq->tunnel; > /* Fill flags for checksum and VLAN. */ > pinfo = vandq_u32(ptype_info, ptype_ol_mask); > pinfo = vreinterpretq_u32_u8( > diff --git a/drivers/net/mlx5/mlx5_rxtx_vec_sse.h > b/drivers/net/mlx5/mlx5_rxtx_vec_sse.h > index c088bcb51..d2492481d 100644 > --- a/drivers/net/mlx5/mlx5_rxtx_vec_sse.h > +++ b/drivers/net/mlx5/mlx5_rxtx_vec_sse.h > @@ -542,6 +542,7 @@ rxq_cq_to_ptype_oflags_v(struct mlx5_rxq_data *rxq, > __m128i cqes[4], > const __m128i mbuf_init = > _mm_loadl_epi64((__m128i *)&rxq->mbuf_initializer); > __m128i rearm0, rearm1, rearm2, rearm3; > + uint8_t pt_idx0, pt_idx1, pt_idx2, pt_idx3; > > /* Extract pkt_info field. */ > pinfo0 = _mm_unpacklo_epi32(cqes[0], cqes[1]); > @@ -595,10 +596,18 @@ rxq_cq_to_ptype_oflags_v(struct mlx5_rxq_data *rxq, > __m128i cqes[4], > /* Errored packets will have RTE_PTYPE_ALL_MASK. */ > op_err = _mm_srli_epi16(op_err, 8); > ptype = _mm_or_si128(ptype, op_err); > - pkts[0]->packet_type = mlx5_ptype_table[_mm_extract_epi8(ptype, 0)]; > - pkts[1]->packet_type = mlx5_ptype_table[_mm_extract_epi8(ptype, 2)]; > - pkts[2]->packet_type = mlx5_ptype_table[_mm_extract_epi8(ptype, 4)]; > - pkts[3]->packet_type = mlx5_ptype_table[_mm_extract_epi8(ptype, 6)]; > + pt_idx0 = _mm_extract_epi8(ptype, 0); > + pt_idx1 = _mm_extract_epi8(ptype, 2); > + pt_idx2 = _mm_extract_epi8(ptype, 4); > + pt_idx3 = _mm_extract_epi8(ptype, 6); > + pkts[0]->packet_type = mlx5_ptype_table[pt_idx0] | > +!!(pt_idx0 & (1 << 6)) * rxq->tunnel; > + pkts[1]->packet_type = mlx5_ptype_table[pt_idx1] | > +!!(pt_idx1 & (1 << 6)) * rxq->tunnel; > + pkts[2]->packet_type = mlx5_ptype_table[pt_idx2] | > +!!(pt_idx2 & (1 << 6)) * rxq->tunnel; > + pkts[3]->packet_type = mlx5_ptype_table[pt_idx3] | > +!!(pt_idx3 & (1 << 6)) * rxq->tunnel; > /* Fill flags for checksum and VLAN. */ > pinfo = _mm_and_si128(pinfo, ptype_ol_mask); > pinfo = _mm_shuffle_epi8(cv_flag_sel, pinfo); > -- > 2.13.3 -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v3 07/14] net/mlx5: support tunnel RSS level

2018-04-13 Thread Nélio Laranjeiro
_fields; > hrxq->tunnel = tunnel; > + hrxq->rss_level = rss_level; > memcpy(hrxq->rss_key, rss_key, rss_key_len); > rte_atomic32_inc(&hrxq->refcnt); > LIST_INSERT_HEAD(&priv->hrxqs, hrxq, next); > @@ -1448,6 +1489,8 @@ mlx5_hrxq_new(struct rte_eth_dev *dev, >

Re: [dpdk-dev] [PATCH v3 08/14] net/mlx5: add hardware flow debug dump

2018-04-13 Thread Nélio Laranjeiro
REMOVE(hrxq, next); > rte_free(hrxq); > diff --git a/drivers/net/mlx5/mlx5_utils.h b/drivers/net/mlx5/mlx5_utils.h > index 85d2aae2b..9a3181b1f 100644 > --- a/drivers/net/mlx5/mlx5_utils.h > +++ b/drivers/net/mlx5/mlx5_utils.h > @@ -103,16 +103,22 @@ extern int mlx5_logtype; > /* claim_zero() does not perform any check when debugging is disabled. */ > #ifndef NDEBUG > > +#define DEBUG(...) DRV_LOG(DEBUG, __VA_ARGS__) > #define claim_zero(...) assert((__VA_ARGS__) == 0) > #define claim_nonzero(...) assert((__VA_ARGS__) != 0) > > #else /* NDEBUG */ > > +#define DEBUG(...) (void)0 > #define claim_zero(...) (__VA_ARGS__) > #define claim_nonzero(...) (__VA_ARGS__) > > #endif /* NDEBUG */ > > +#define INFO(...) DRV_LOG(INFO, __VA_ARGS__) > +#define WARN(...) DRV_LOG(WARNING, __VA_ARGS__) > +#define ERROR(...) DRV_LOG(ERR, __VA_ARGS__) > + > /* Convenience macros for accessing mbuf fields. */ > #define NEXT(m) ((m)->next) > #define DATA_LEN(m) ((m)->data_len) > -- > 2.13.3 This is a really good first step, even if you could also spread the user of the DEBUG macro all over the Verbs object creations, they are purely developer needs. Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v3 09/14] net/mlx5: introduce VXLAN-GPE tunnel type

2018-04-13 Thread Nélio Laranjeiro
rxtx.c > index 285b2dbf0..c9342d659 100644 > --- a/drivers/net/mlx5/mlx5_rxtx.c > +++ b/drivers/net/mlx5/mlx5_rxtx.c > @@ -466,8 +466,7 @@ mlx5_tx_burst(void *dpdk_txq, struct rte_mbuf **pkts, > uint16_t pkts_n) > uint8_t vlan_sz = > (buf->ol_flags & PKT_TX_VLAN_PKT) ? 4 : 0; > const uint64_t is_tunneled = > - buf->ol_flags & (PKT_TX_TUNNEL_GRE | > - PKT_TX_TUNNEL_VXLAN); > + buf->ol_flags & (PKT_TX_TUNNEL_MASK); > > tso_header_sz = buf->l2_len + vlan_sz + > buf->l3_len + buf->l4_len; > -- > 2.13.3 > Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v3 11/14] net/mlx5: support MPLS-in-GRE and MPLS-in-UDP

2018-04-13 Thread Nélio Laranjeiro
ef HAVE_IBV_DEVICE_MPLS_SUPPORT > unsigned int size = sizeof(struct ibv_flow_spec_tunnel); > struct ibv_flow_spec_tunnel tunnel = { > .type = parser->inner | IBV_FLOW_SPEC_VXLAN_TUNNEL, > .size = size, > }; > +#else > + const struct rte_flow_item_gre *spec = item->spec; > + const struct rte_flow_item_gre *mask = item->mask; > + unsigned int size = sizeof(struct ibv_flow_spec_gre); > + struct ibv_flow_spec_gre tunnel = { > + .type = parser->inner | IBV_FLOW_SPEC_GRE, > + .size = size, > + }; > +#endif > > parser->inner = IBV_FLOW_SPEC_INNER; > parser->tunnel = ptype_ext[PTYPE_IDX(RTE_PTYPE_TUNNEL_GRE)]; > parser->out_layer = parser->layer; > parser->layer = HASH_RXQ_TUNNEL; > +#ifdef HAVE_IBV_DEVICE_MPLS_SUPPORT > + if (spec) { > + if (!mask) > + mask = default_mask; > + tunnel.val.c_ks_res0_ver = spec->c_rsvd0_ver; > + tunnel.val.protocol = spec->protocol; > + tunnel.val.c_ks_res0_ver = mask->c_rsvd0_ver; > + tunnel.val.protocol = mask->protocol; > + /* Remove unwanted bits from values. */ > + tunnel.val.c_ks_res0_ver &= tunnel.mask.c_ks_res0_ver; > + tunnel.val.protocol &= tunnel.mask.protocol; > + tunnel.val.key &= tunnel.mask.key; > + } > +#endif > mlx5_flow_create_copy(parser, &tunnel, size); > return 0; > } > -- > 2.13.3 > Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v3 12/14] doc: update mlx5 guide on tunnel offloading

2018-04-13 Thread Nélio Laranjeiro
et. > -- Hardware checksum RX offloads for VXLAN inner header are not supported yet. > - For secondary process: > >- Forked secondary process not supported. > -- > 2.13.3 As discussed in [1], also add a new entry in the features list, Thanks, [1] https://dpdk.org/ml/archives/dev/2018-April/096736.html -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v3 13/14] net/mlx5: fix invalid flow item check

2018-04-13 Thread Nélio Laranjeiro
ret = -ENOTSUP; > goto exit_item_not_supported; > + } > cur_item = token; > ret = mlx5_flow_item_validate(items, > (const uint8_t *)cur_item->mask, > -- > 2.13.3 Should be proposed outside of this series. Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v3 14/14] net/mlx5: support RSS configuration in isolated mode

2018-04-13 Thread Nélio Laranjeiro
are never created to comply with the isolated mode. There is no need to enable such API. Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v3 01/14] net/mlx5: support 16 hardware priorities

2018-04-13 Thread Nélio Laranjeiro
On Fri, Apr 13, 2018 at 01:10:07PM +, Xueming(Steven) Li wrote: > > > > -Original Message- > > From: Nélio Laranjeiro > > Sent: Friday, April 13, 2018 7:59 PM > > To: Xueming(Steven) Li > > Cc: Shahaf Shuler ; dev@dpdk.org > > Subject:

Re: [dpdk-dev] [PATCH v3 11/14] net/mlx5: support MPLS-in-GRE and MPLS-in-UDP

2018-04-13 Thread Nélio Laranjeiro
On Fri, Apr 13, 2018 at 02:48:17PM +, Xueming(Steven) Li wrote: > > > > -Original Message- > > From: Nélio Laranjeiro > > Sent: Friday, April 13, 2018 9:37 PM > > To: Xueming(Steven) Li > > Cc: Shahaf Shuler ; dev@dpdk.org > > Subject: Re:

Re: [dpdk-dev] [PATCH v2 07/15] net/mlx5: support tunnel RSS level

2018-04-16 Thread Nélio Laranjeiro
->ctx, > > > + &(struct ibv_qp_init_attr_ex){ > > > + .qp_type = IBV_QPT_RAW_PACKET, > > > + .comp_mask = > > > + IBV_QP_INIT_ATTR_PD | > > > + IBV_QP_INIT_ATTR_IND_TABLE | > > > + IBV_QP_INIT_ATTR_RX_HASH, > > > + .rx_hash_conf = (struct ibv_rx_hash_conf){ > > > + .rx_hash_function = IBV_RX_HASH_FUNC_TOEPLITZ, > > > + .rx_hash_key_len = rss_key_len, > > > + .rx_hash_key = (void *)(uintptr_t)rss_key, > > > + .rx_hash_fields_mask = hash_fields | > > > + (tunnel && rss_level ? > > > + (uint32_t)IBV_RX_HASH_INNER : 0), >[...] .rx_hash_fields_mask = hash_fields | (rss_level > 1) ? (uint32_t)IBV_RX_HASH_INNER : 0), Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v3 04/14] net/mlx5: support Rx tunnel type identification

2018-04-16 Thread Nélio Laranjeiro
On Sat, Apr 14, 2018 at 12:57:58PM +, Xueming(Steven) Li wrote: > +Adrien > > > -Original Message- > > From: Nélio Laranjeiro > > Sent: Friday, April 13, 2018 9:03 PM > > To: Xueming(Steven) Li > > Cc: Shahaf Shuler ; dev@dpdk.org; Olivier Matz &

Re: [dpdk-dev] [PATCH v2 07/15] net/mlx5: support tunnel RSS level

2018-04-16 Thread Nélio Laranjeiro
eems to be redundant here. > > > > > > > > rss_level > 1 is equivalent to tunnel, there is no need to have both. > > > > > > There is a case of tunnel and outer rss(1). > > > > Why cannot it be handled by a regular Hash Rx queue, i.e. what

Re: [dpdk-dev] [PATCH v3 11/14] net/mlx5: support MPLS-in-GRE and MPLS-in-UDP

2018-04-16 Thread Nélio Laranjeiro
cur_item = &mlx5_flow_items[items->type]; > ret = cur_item->convert(items, > (cur_item->default_mask ? >cur_item->default_mask : >cur_item->mask), >&data); > > This code is after the mlx5_flow_convert_items_validate() which refuses unknown items, if you you see an unknown item reaching this code above, there is bug somewhere and it should be fixed. Un-supported items should not be in the static array. Regards, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v3 07/14] net/mlx5: support tunnel RSS level

2018-04-16 Thread Nélio Laranjeiro
ll more annoying than helping. Example: flow create 0 ingress eth ethertype is 0x0806 / end actions rss .... won't raise such log, whereas ARP is not an IP protocol and thus can be RSS'ed. flow create 0 ingress eth / ipv4 / end actions rss type ipv6... will raise the log, but it is obvious the user won't have RSS. Regards, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v2 07/15] net/mlx5: support tunnel RSS level

2018-04-16 Thread Nélio Laranjeiro
On Mon, Apr 16, 2018 at 10:06:06AM +, Xueming(Steven) Li wrote: > > > > -Original Message- > > From: Nélio Laranjeiro > > Sent: Monday, April 16, 2018 4:09 PM > > To: Xueming(Steven) Li > > Cc: Shahaf Shuler ; dev@dpdk.org > > Subject: Re:

Re: [dpdk-dev] [PATCH] net/mlx5: fix parsing all-multicast from flow item

2018-01-11 Thread Nélio Laranjeiro
ill not work i.e. 0xff != 0x01 but it as the multicast bit set. From my understanding, Verbs flow attribute must also be modified in such situation. Are you sure about this change? Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH 1/4] net/mlx5: revert "fix Memory Region registration"

2018-01-15 Thread Nélio Laranjeiro
EN(mb)); > + uintptr_t addr = rte_pktmbuf_mtod(mb, uintptr_t); > struct mlx5_mr *mr; > > assert(i < RTE_DIM(txq->mp2mr)); > -- > 2.13.3 > Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH 2/4] net/mlx5: forbid MR registration in secondary process

2018-01-15 Thread Nélio Laranjeiro
assert(rte_cpu_to_be_32(txq->mp2mr[i]->mr->lkey) == > -txq->mp2mr[i]->lkey); > txq->mr_cache_idx = i; > return txq->mp2mr[i]->lkey; > } > -- > 2.13.3 > Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH 3/4] net/mlx5: support mempool of multi-mem segments

2018-01-15 Thread Nélio Laranjeiro
add a new MR for it. */ > + if (txq->mp2mr[i] == NULL) > break; > - } > + if (i == txq->mr_cache_idx) > + continue; > if (txq->mp2mr[i]->start <= addr && > txq->mp2mr[i]->end >= addr) { > +#ifndef NDEBUG > + if (rte_eal_process_type() == RTE_PROC_PRIMARY) > + assert(rte_cpu_to_be_32(txq->mp2mr[i]->mr->lkey) > +== txq->mp2mr[i]->lkey); > +#endif > txq->mr_cache_idx = i; > return txq->mp2mr[i]->lkey; > } > } > - txq->mr_cache_idx = 0; > - mr = mlx5_txq_mp2mr_reg(txq, mlx5_tx_mb2mp(mb), i); > - /* > - * Request the reference to use in this queue, the original one is > - * kept by the control plane. > - */ > - if (mr) { > - rte_atomic32_inc(&mr->refcnt); > - return mr->lkey; > + mr = mlx5_mp2mr(container_of(txq, struct mlx5_txq_ctrl, txq)->priv, > + mlx5_tx_mb2mp(mb), addr); > + assert(mr); > + /* recent used MR first */ > + if (i) { > + if (i >= RTE_DIM(txq->mp2mr)) { > + i = RTE_DIM(txq->mp2mr) - 1; > + DEBUG("TXQ MR cache full, please consider increasing > CONFIG_RTE_LIBRTE_MLX5_TX_MP_CACHE in config file"); Please respect the coding style of the PMD/file split those lines. > + } > + memmove(&txq->mp2mr[1], &txq->mp2mr[0], > + i * sizeof(sizeof(txq->mp2mr[0]))); > } > - return (uint32_t)-1; > + /* Store the new entry. */ > + txq->mp2mr[0] = mr; > + txq->mr_cache_idx = 0; > + return mr ? mr->lkey : (uint32_t)-1; > } > > /** > diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c > index 1a20967a2..920cc0f46 100644 > --- a/drivers/net/mlx5/mlx5_trigger.c > +++ b/drivers/net/mlx5/mlx5_trigger.c > @@ -58,17 +58,10 @@ priv_txq_start(struct priv *priv) > > /* Add memory regions to Tx queues. */ > for (i = 0; i != priv->txqs_n; ++i) { > - unsigned int idx = 0; > - struct mlx5_mr *mr; > struct mlx5_txq_ctrl *txq_ctrl = mlx5_priv_txq_get(priv, i); > > if (!txq_ctrl) > continue; > - LIST_FOREACH(mr, &priv->mr, next) { > - priv_txq_mp2mr_reg(priv, &txq_ctrl->txq, mr->mp, idx++); > - if (idx == MLX5_PMD_TX_MP_CACHE) > - break; > - } > txq_alloc_elts(txq_ctrl); > txq_ctrl->ibv = mlx5_priv_txq_ibv_new(priv, i); > if (!txq_ctrl->ibv) { > -- > 2.13.3 > Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH 4/4] net/mlx5: remove MR refcnt

2018-01-16 Thread Nélio Laranjeiro
gt; (void *)ctrl, rte_atomic32_read(&ctrl->refcnt)); > @@ -834,7 +823,6 @@ mlx5_priv_txq_get(struct priv *priv, uint16_t idx) > int > mlx5_priv_txq_release(struct priv *priv, uint16_t idx) > { > - unsigned int i; > struct mlx5_txq_ctrl *txq; > > if (!(*priv->txqs)[idx]) > @@ -849,12 +837,6 @@ mlx5_priv_txq_release(struct priv *priv, uint16_t idx) > if (!ret) > txq->ibv = NULL; > } > - for (i = 0; i != MLX5_PMD_TX_MP_CACHE; ++i) { > - if (txq->txq.mp2mr[i]) { > - priv_mr_release(priv, txq->txq.mp2mr[i]); > - txq->txq.mp2mr[i] = NULL; > - } > - } Nullifying the case when the queue is release should remain. It is better to detect an access to a null pointer than to a freed memory space. > if (rte_atomic32_dec_and_test(&txq->refcnt)) { > txq_free_elts(txq); > LIST_REMOVE(txq, next); > -- > 2.13.3 > Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH] net/mlx5: fix handling link status change event

2018-01-16 Thread Nélio Laranjeiro
callbacks > + * accordingly. > * > * @param dev > * Pointer to Ethernet device structure. > @@ -893,17 +935,54 @@ mlx5_link_update_unlocked_gs(struct rte_eth_dev *dev, > int wait_to_complete) > * Wait for request completion (ignored). > */ > int > -mlx5_link_update(struct rte_eth_dev *dev, int wait_to_complete) > +priv_link_update(struct priv *priv, int wait_to_complete) Seems to be also the case here. Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH 1/2] net: fix ESP header byte ordering definition

2018-01-16 Thread Nélio Laranjeiro
ch before fixing the code using it. > > Cc: bor...@mellanox.com > > > > Signed-off-by: Nelio Laranjeiro > > > > [1] https://tools.ietf.org/html/rfc2406 > > Acked-by: Olivier Matz -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v2] net/mlx5: fix handling link status change event

2018-01-17 Thread Nélio Laranjeiro
it isn't > effective. It is better to change burst functions when link status change > is detected. > > Fixes: 62072098b54e ("mlx5: support setting link up or down") > Cc: sta...@dpdk.org > > Signed-off-by: Yongseok Koh Acked-by: Nelio Laranjeiro Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH 1/2] net/mlx5: fix return value of start operation

2018-01-18 Thread Nélio Laranjeiro
error; > } > err = priv_rx_intr_vec_enable(priv); > @@ -196,7 +198,7 @@ mlx5_dev_start(struct rte_eth_dev *dev) > priv_rxq_stop(priv); > priv_flow_delete_drop_queue(priv); > priv_unlock(priv); > - return -err; > + return err; > } > > /** err in the function is handled with positives errno's, adding only those two and returning err will make the other positive. Regards, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH 2/2] net/mlx5: fix allocation when no memory on device NUMA node

2018-01-18 Thread Nélio Laranjeiro
; This function is the finalisation of the creation of the queues and contains the buffers to the CQ/WQ which must be on the correct socket otherwise the performances will be limited. Even if this function is only called on dev_start() it must reflect the configuration requested from the application on the rte_eth_{tx,rx}_queue_setup(). Regards, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH 1/2] net/mlx5: fix return value of start operation

2018-01-19 Thread Nélio Laranjeiro
On Thu, Jan 18, 2018 at 05:13:08PM +0100, Olivier Matz wrote: > On Thu, Jan 18, 2018 at 05:04:27PM +0100, Nélio Laranjeiro wrote: > > On Thu, Jan 18, 2018 at 02:00:42PM +0100, Olivier Matz wrote: > > > On error, mlx5_dev_start() does not return a negative value > > > a

Re: [dpdk-dev] [PATCH] net/mlx5: fix Memory Region lookup

2018-01-19 Thread Nélio Laranjeiro
Cc: sta...@dpdk.org > > Reported-by: Xueming Li > Signed-off-by: Xueming Li > Signed-off-by: Yongseok Koh Acked-by: Nelio Laranjeiro -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH 1/2] net/mlx5: fix return value of start operation

2018-01-19 Thread Nélio Laranjeiro
On Fri, Jan 19, 2018 at 02:30:45PM +0100, Olivier Matz wrote: > On Fri, Jan 19, 2018 at 09:43:14AM +0100, Olivier Matz wrote: > > On Fri, Jan 19, 2018 at 09:35:01AM +0100, Nélio Laranjeiro wrote: > > > On Thu, Jan 18, 2018 at 05:13:08PM +0100, Olivier Matz wrote: > > > &

Re: [dpdk-dev] [PATCH 1/2] net/mlx5: fix return value of start operation

2018-01-19 Thread Nélio Laranjeiro
On Fri, Jan 19, 2018 at 02:30:45PM +0100, Olivier Matz wrote: > On Fri, Jan 19, 2018 at 09:43:14AM +0100, Olivier Matz wrote: > > On Fri, Jan 19, 2018 at 09:35:01AM +0100, Nélio Laranjeiro wrote: > > > On Thu, Jan 18, 2018 at 05:13:08PM +0100, Olivier Matz wrote: > > > &

Re: [dpdk-dev] [PATCH] net/mlx5: remmap UAR address for multiple process

2018-01-22 Thread Nélio Laranjeiro
eturn -ENOMEM; > + } > + /* Accept either same addr or a new addr returned from mmap if target > + * range occupied. > + */ > + INFO("Reserved UAR address space: 0x%p", addr); The '%p' already prefix the address with the 0x. > + priv->uar_base = addr; /* for primary and secondary UAR re-mmap */ > + uar_base = addr; /* process local, don't reserve again */ > + return 0; > +} > + Regards, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH] net/mlx5: use PCI BDF as the port name

2018-01-23 Thread Nélio Laranjeiro
lx5, mrvl, softnic don't. Why not addressing all drivers at once? > Signed-off-by: Yuanhan Liu Acked-by: Nelio Laranjeiro -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH] net/mlx5: remmap UAR address for multiple process

2018-01-23 Thread Nélio Laranjeiro
lio, > > > -Original Message- > > From: Nélio Laranjeiro [mailto:nelio.laranje...@6wind.com] > > Sent: Monday, January 22, 2018 10:53 PM > > To: Xueming(Steven) Li > > Cc: Shahaf Shuler ; dev@dpdk.org > > Subject: Re: [PATCH] net/mlx5: remmap UAR addres

Re: [dpdk-dev] [PATCH] net/mlx5: fix link state on device start

2018-01-24 Thread Nélio Laranjeiro
ot;Failed to set port %u link to be up", > + dev->data->port_id); > + goto error; > + } > priv_dev_interrupt_handler_install(priv, dev); > priv_unlock(priv); > return 0; > -- > 2.12.0 According to mlx5_dev_start() documentation function: * @return * 0 on success, negative errno value on failure. This code is returning -1 in case of error, which means: EPERM 1 /* Operation not permitted */ which is a wrong value. Why not returning an errno in your priv function with an EBUSY or EAGAIN which is more accurate? Regards, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH 2/5] net/mlx5: fix secondary process mempool registration

2018-01-24 Thread Nélio Laranjeiro
reate mempool before rte_eth_dev_start()", > + (void *)mp, mp->name); > + assert(rte_eal_process_type() == RTE_PROC_PRIMARY); > + return (uint32_t)-1; > + } > + mr = mlx5_txq_mp2mr_reg(txq, mp, i); > /* >* Request the reference to use in this queue, the original one is >* kept by the control plane. > -- > 2.12.0 -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH 3/5] net/mlx5: assert for un-successful memory registration

2018-01-24 Thread Nélio Laranjeiro
(void *)mp, mp->name); > + assert(mr != NULL); This assert seems wrong. Why this assert, you don't trust the CPU to verify the pointer is NULL? > } > return (uint32_t)-1; > } > -- > 2.12.0 Regards, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v2] net/mlx5: mmap uar address around huge pages

2018-01-25 Thread Nélio Laranjeiro
Xueming Li As it does not break anything... Acked-by: Nelio Laranjeiro -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH 5/5] net/mlx5: remove old MLNX_OFED 3.3 verification

2017-08-22 Thread Nélio Laranjeiro
Nelio Laranjeiro > > Acked-by: Shahaf Shuler Hi Ferruh, This is already documented in the NIC guides, see section "20.5. Prerequisites" [1]. Thanks, [1] http://dpdk.org/doc/guides/nics/mlx5.html -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH 4/5] net/mlx5: remove multiple drop RSS queues

2017-08-22 Thread Nélio Laranjeiro
site. So the user knows which version of MLNX OFED is installed in his system. > > Signed-off-by: Nelio Laranjeiro > > Acked-by: Shahaf Shuler > > <...> Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH 3/5] net/mlx5: fix non working secondary process by removing it

2017-08-22 Thread Nélio Laranjeiro
esses") > > > > Signed-off-by: Nelio Laranjeiro > > Acked-by: Shahaf Shuler > > <...> I'll update the feature documentation in a v2 to remove the "Multiprocess aware" from the feature list in this patch. It will be added back with the new implementation. Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH 2/5] net/mlx5: remove pdentic pragma

2017-08-22 Thread Nélio Laranjeiro
gt; > > > > -/* DPDK headers don't like -pedantic. */ > > -#ifdef PEDANTIC > > -#pragma GCC diagnostic ignored "-Wpedantic" > > -#endif > > Good to see that they are going, thanks. Is verbs.h one still required? Yes it is, Adrien is pushing some patch to the OFED community to make it also pedantic. Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v1 00/21] net/mlx5: cleanup for isolated mode

2017-08-22 Thread Nélio Laranjeiro
t; Patchset is not applying cleanly, can you please rebase it on top of > latest tree? yes, > And there are some checkpatch warnings for the set. sure, > There are two other mlx5 patchsets, what is the dependency between them? It is mlx5-cleanup first and then this one. I will put clearly the dependency in cover letter to help. Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH 1/2] net/mlx5: support device removal event

2017-08-23 Thread Nélio Laranjeiro
- ret = priv_dev_link_status_handler(priv, dev); > + ret = priv_link_status_alarm_update(priv); It is not clear, this calls an alarm_update without getting the link status, the function name is "link_status_handler" why does the behavior does not reflect the function name? It is too confusing to be integrated as is, we had several bugs in this part of the code, keep it clear, by keeping the old functions name. Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH 2/2] net/mlx5: fix probe failure report

2017-08-23 Thread Nélio Laranjeiro
dapters") > Fixes: 1371f4df16bc ("mlx5: check port is configured as ethernet device") > > Signed-off-by: Matan Azrad > Cc: sta...@dpdk.org Acked-by: Nelio Laranjeiro -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH 1/2] net/mlx5: replace memory barrier type

2017-08-23 Thread Nélio Laranjeiro
; /* Ensure ordering between DB record and BF copy. */ > rte_wmb(); > -- > 2.7.4 Acked-by: Nelio Laranjeiro -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH 2/2] net/mlx5: don't map doorbell register to write combining

2017-08-23 Thread Nélio Laranjeiro
ment variable are read by libmlx5 code and the PMD is linked though libibverbs which does not provide a "clean" way tweak libmlx5. It is not used to communicate with another process, it is the same process executing the code which reads those environment variables. Currently it is the only solution we have. For the patch: Acked-by: Nelio Laranjeiro Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH 3/3] net/mlx5: add hardware timestamp

2017-08-23 Thread Nélio Laranjeiro
using it > will disable the cqe comprission, and will add extra checkes in > the vec rx path. > > Signed-off-by: Raslan Darawsheh The new parameter should be documented in the NIC documentation. Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH] net/mlx5: fix xstats functions unlock missing

2017-08-23 Thread Nélio Laranjeiro
(priv); > if (stats_n < 0) > - return; > + goto unlock; > if (xstats_ctrl->stats_n != stats_n) > priv_xstats_init(priv); > priv_xstats_reset(priv); > +unlock: > priv_unlock(priv); > } > > -- > 2.7.4 > Acked-by: Nelio Laranjeiro -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [RFC] net/mlx5: support count flow action

2017-08-23 Thread Nélio Laranjeiro
NULL, > "Flow count unsupported"); > + (void)dev; > + (void)flow; > + (void)type; > + (void)res; > + (void)error; > + return -1; Sa

Re: [dpdk-dev] [PATCH 1/2] net/mlx5: support device removal event

2017-08-24 Thread Nélio Laranjeiro
On Wed, Aug 23, 2017 at 07:44:45PM +, Matan Azrad wrote: > Hi Nelio > > > -Original Message- > > From: Nélio Laranjeiro [mailto:nelio.laranje...@6wind.com] > > Sent: Wednesday, August 23, 2017 12:41 PM > > To: Matan Azrad > > Cc: Adrien Maza

Re: [dpdk-dev] [PATCH] net/mlx5: extend debug logs verbosity

2017-08-24 Thread Nélio Laranjeiro
; --- > drivers/net/mlx5/mlx5_rxtx.h | 20 +--- > 1 file changed, 17 insertions(+), 3 deletions(-) Acked-by: Nelio Laranjeiro -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH 1/2] net/mlx5: fix num seg assumption on vPMD

2017-08-24 Thread Nélio Laranjeiro
eg(&pkts[nb_tx], n); > + if (!n) > + break; > if (!(txq->flags & ETH_TXQ_FLAGS_NOOFFLOADS)) > n = txq_calc_offload(txq, &pkts[nb_tx], n, &cs_flags); > ret = txq_burst_v(txq, &pkts[nb_tx], n, cs_flags); > -- > 2.12.0 Acked-by: Nelio Laranjeiro -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH 2/2] net/mlx5: enforce Tx num of segments limitation

2017-08-24 Thread Nélio Laranjeiro
h > @@ -154,6 +154,9 @@ > /* Default mark value used when none is provided. */ > #define MLX5_FLOW_MARK_DEFAULT 0xff > > +/* Maximum number of DS in WQE. */ > +#define MLX5_MAX_DS (63) > + Why the parenthesis? Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v2 1/3] ethdev: expose Rx hardware timestamp

2017-08-24 Thread Nélio Laranjeiro
efine DEV_RX_OFFLOAD_QINQ_STRIP 0x0020 > #define DEV_RX_OFFLOAD_OUTER_IPV4_CKSUM 0x0040 > #define DEV_RX_OFFLOAD_MACSEC_STRIP 0x0080 > +#define DEV_RX_OFFLOAD_TIMESTAMP 0x0100 > +/**< Device puts raw timestamp in mbuf. */ > > /** > * TX offload capabilities of a device. > -- > 2.7.4 -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v2 2/3] app/testpmd: add Rx timestamp in testpmd

2017-08-24 Thread Nélio Laranjeiro
if (ol_flags & PKT_RX_TIMESTAMP) > + printf(" - timestamp %lu ", mb->timestamp); > if (ol_flags & PKT_RX_VLAN_STRIPPED) > printf(" - VLAN tci=0x%x", mb->vlan_tci); > if (ol_flags & PKT_RX_QINQ_STRIPPED) > -- > 2.7.4 How can we enable this Rx offload? Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v2 3/3] net/mlx5: add hardware timestamp

2017-08-24 Thread Nélio Laranjeiro
+++- > 8 files changed, 53 insertions(+), 3 deletions(-) It is strange to enable/disable this single offload again the application request. Why do you need such behavior? Another point I don't seen any code to retrieve this offloads request from the port configuration as for the ot

Re: [dpdk-dev] [PATCH v1] net/mlx5: support upstream rdma-core

2017-08-24 Thread Nélio Laranjeiro
; + wq[i].addr = rte_cpu_to_be_64((uintptr_t)elts[i]->buf_addr + > RTE_PKTMBUF_HEADROOM); Same here this should be in another commit. > rxq->rq_ci += n; > rte_wmb(); > diff --git a/drivers/net/mlx5/mlx5_txq.c b/drivers/net/mlx5/mlx5_txq.c > index 4b0b532..3156ad2 100644 > --- a/drivers/net/mlx5/mlx5_txq.c > +++ b/drivers/net/mlx5/mlx5_txq.c > @@ -241,16 +247,16 @@ > if (priv->mps == MLX5_MPW_ENHANCED) > tmpl.txq.mpw_hdr_dseg = priv->mpw_hdr_dseg; > /* MRs will be registered in mp2mr[] later. */ > - attr.cq = (struct ibv_exp_cq_init_attr){ > + attr.cq = (struct ibv_cq_init_attr_ex){ > .comp_mask = 0, > }; > cqe_n = ((desc / MLX5_TX_COMP_THRESH) - 1) ? > ((desc / MLX5_TX_COMP_THRESH) - 1) : 1; > if (priv->mps == MLX5_MPW_ENHANCED) > cqe_n += MLX5_TX_COMP_THRESH_INLINE_DIV; > - tmpl.cq = ibv_exp_create_cq(priv->ctx, > + tmpl.cq = ibv_create_cq(priv->ctx, > cqe_n, > - NULL, NULL, 0, &attr.cq); > + NULL, NULL, 0); Indentation issue. Please split this in the three commits: - first fixing the return value, - one changing the hton/ntoh by rte equivalents, - and this one. Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [RFC] net/mlx5: support count flow action

2017-08-24 Thread Nélio Laranjeiro
Hi Ori, On Thu, Aug 24, 2017 at 02:04:32PM +, Ori Kam wrote: > Hi Nelio, > > Please see my comments in line. > > Ori > > > -Original Message- > > From: Nélio Laranjeiro [mailto:nelio.laranje...@6wind.com] > > Sent: Thursday, August 24,

Re: [dpdk-dev] [PATCH v1 1/2] net/mlx5: change eth device reference for secondary process

2017-08-24 Thread Nélio Laranjeiro
dev->data->dev_private; > > + if (mlx5_is_secondary()) > + return -E_RTE_SECONDARY; > + Extra empty line also in the following copy/past of this if statement. >[...] Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v1 2/2] net/mlx5: add multiple process support

2017-08-25 Thread Nélio Laranjeiro
d not make such assumption. > - if (priv == NULL) > - return 0; > - primary_priv = > - mlx5_secondary_data[priv->dev->data->port_id].primary_priv; > - /* Look for queue index in both private structures. */ > - for (index = 0; index != priv->txqs_n; ++index) > - if (((*primary_priv->txqs)[index] == txq) || > - ((*priv->txqs)[index] == txq)) > - break; > - if (index == priv->txqs_n) > - return 0; > - txq = (*priv->txqs)[index]; > - return priv->dev->tx_pkt_burst(txq, pkts, pkts_n); > + for (i = 0; i != priv->txqs_n; ++i) { > + txq = (*priv->txqs)[i]; > + txq_ctrl = container_of(txq, struct txq_ctrl, txq); > + uar_va = (uintptr_t)txq_ctrl->txq.bf_reg; > + uar_va = RTE_ALIGN_FLOOR(uar_va, > + page_size / priv->num_uars_per_page); Indentation. > + already_mapped = 0; > + for (j = 0; j != pages_n; ++j) { > + if (pages[j] == uar_va) { > + already_mapped = 1; > + break; > + } > + } > + if (already_mapped) > + continue; > + Extra empty line. > + pages[pages_n++] = uar_va; > + addr = mmap((void *)uar_va, page_size, > + PROT_WRITE, MAP_FIXED | MAP_SHARED, fd, > + txq_ctrl->uar_mmap_offset); Indentation. > + if (addr != (void *)uar_va) { > + ERROR("call to mmap failed on UAR for txq %d\n", i); > + return -1; > + } > + } > + return 0; > } > -- > 1.8.3.1 Nice work. Thanks, [1] http://dpdk.org/ml/archives/dev/2017-August/073212.html -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v1 1/2] net/mlx5: change eth device reference for secondary process

2017-08-25 Thread Nélio Laranjeiro
> > be renamed to: > > priv_dev_select_rx/tx_function(struct *priv, struct rte_eth_dev *dev, ...) > > > > this will avoid the multiple lock/unlocks inside the functions. > So priv_* are lock-free functions? priv_*() assume the lock have been done by the caller. Hope it helps. Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH 1/2] net/mlx5: support device removal event

2017-08-25 Thread Nélio Laranjeiro
n > In this function description or internal comment for future code review. > If you want it, please suggest comment. Yes the comment can added internally. Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [RFC PATCH 1/1] net/mlx5: add vectorized Rx/Tx burst for ARM

2017-09-04 Thread Nélio Laranjeiro
Hi Yongseok, Some questions/comments below, On Fri, Aug 25, 2017 at 11:40:23AM -0700, Yongseok Koh wrote: > New Rx/Tx burst functions are added using NEON vector instructions for ARM > CPU. > > Signed-off-by: Yongseok Koh > --- > drivers/net/mlx5/Makefile |2 + > drivers/net/ml

Re: [dpdk-dev] [PATCH v2] net/mlx5: support device removal event

2017-09-04 Thread Nélio Laranjeiro
R_LSC, NULL, > - NULL); > + NULL); Same as above, why is this line modified? > } > > /** > @@ -1192,14 +1228,17 @@ mlx5_dev_interrupt_handler(void *cb_arg) > { > struct rte_eth_dev *dev = cb_arg; > struct priv *priv = dev->data->dev_private; > - int ret; > + uint32_t events; > > priv_lock(priv); > - ret = priv_dev_link_status_handler(priv, dev); > + events = priv_dev_status_handler(priv); > priv_unlock(priv); > - if (ret) > + if (events & (1 << RTE_ETH_EVENT_INTR_LSC)) > _rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_INTR_LSC, NULL, > - NULL); > + NULL); Same question here, > + if (events & (1 << RTE_ETH_EVENT_INTR_RMV)) > + _rte_eth_dev_callback_process(dev, RTE_ETH_EVENT_INTR_RMV, NULL, > + NULL); and here. Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH] net/mlx5: fix calculating TSO inline size

2017-09-04 Thread Nélio Laranjeiro
f (copy_b && ((end - (uintptr_t)raw) > copy_b)) { > /* >* One Dseg remains in the current WQE. To Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v2] net/mlx5: support upstream rdma-core

2017-09-04 Thread Nélio Laranjeiro
> This patch should be applied after the series: > http://dpdk.org/dev/patchwork/patch/28325/ > > Ack. > -- > Nélio Laranjeiro > 6WIND I am pretty sure I did not acked this commit log written like this. Please send a v3 with a correct commit log written. I'll send the ack myself. Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v2 2/2] net/mlx5: enforce Tx num of segments limitation

2017-09-04 Thread Nélio Laranjeiro
uch informations, but it will reduce considerably the throughput in some cases where the issue is not related to this. Datapath must also be optimised also in debug mode. I suggestion would be to change the burst API to return an int with correct errors values. Currently oerrors can be increased in such situation to provide more information to the user. Thanks, -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH v2 1/2] net/mlx5: fix num seg assumption on vPMD

2017-09-04 Thread Nélio Laranjeiro
_t > pkts_n) > > if (!pkts_n) > return 0; > - assert(NB_SEGS(pkts[0]) == 1); > /* Count the number of continuous single segment packets. */ > - for (pos = 1; pos < pkts_n; ++pos) > + for (pos = 0; pos < pkts_n; ++pos) >

Re: [dpdk-dev] [PATCH] net/mlx5: fix tunnel offload detection

2017-09-04 Thread Nélio Laranjeiro
nel_en = 1; > mps = MLX5_MPW; > break; > case PCI_DEVICE_ID_MELLANOX_CONNECTX5: > -- > 1.8.3.1 Acked-by: Nelio Laranjeiro -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH 1/3] net/mlx5: replace network to host macros

2017-09-04 Thread Nélio Laranjeiro
t; > Signed-off-by: Shachar Beiser -- Nélio Laranjeiro 6WIND

Re: [dpdk-dev] [PATCH 2/3] net/mlx5: fix TSO MLNX OFED 3.3 verification

2017-09-04 Thread Nélio Laranjeiro
EMPW_MAX_INLINE_LEN (4U * MLX5_WQE_SIZE) > > -#ifndef HAVE_VERBS_MLX5_OPCODE_TSO > -#define MLX5_OPCODE_TSO MLX5_OPCODE_LSO_MPW /* Compat with OFED 3.3. */ > -#endif > > #define MLX5_OPC_MOD_ENHANCED_MPSW 0 > #define MLX5_OPCODE_ENHANCED_MPSW 0x29 > -- > 1.8.3.1 > Acked-by: Nelio Laranjeiro -- Nélio Laranjeiro 6WIND

<    1   2   3   4   5   >