MQ RX points several queues at the same adapter-wide counters, which
races the updates and leaves no way to attribute a count to a queue.
The patch has three parts:
- storage: every counter with more than one writer moves into
per-queue structs, allocated at probe and freed at remove, with
a single writer per slot.
- export: packets, bytes and hardware drops are reported per queue
through netdev_stat_ops; ndo_get_stats64() sums every allocated
slot (live and retired after shrink) into the device totals;
ethtool -S keeps its thirteen keys (summed on read) and adds a few
driver-specific per-queue keys.
- rx_no_buffer: each queue carries the drops from buffer-list
pages it has already released, so the count is cumulative across
a reopen and never goes backwards.
The per-queue structs are:
struct ibmveth_rx_queue_stats
struct ibmveth_tx_queue_stats
Each slot has a single writer -- replenish_* under that queue's
replenish_lock, interrupts from that queue's IRQ handler, the
other RX fields from its NAPI (polls is not counted for
netpoll's budget-0 call), TX under
the stack's per-queue TX lock -- so u64 needs no atomic and no
u64_stats_sync on this PPC64-only driver. Readers run locklessly on
other CPUs, so writers go through WRITE_ONCE() (IBMVETH_STAT_INC/ADD)
and readers use READ_ONCE(), which keeps KCSAN quiet and stops the
compiler from splitting or refetching the accesses.
packets, bytes and hardware RX drops go through struct
netdev_stat_ops (.get_queue_stats_rx, .get_queue_stats_tx,
.get_base_stats). rx-hw-drops is what PHYP dropped: its no-buffer
count and the slots it hands back marked invalid. The no-buffer count
alone is also rx-hw-drop-overruns. rx-alloc-fail is the replenish
skb allocation failures (replenish_no_mem). Driver software drops (bad
correlator, failed harvest, oversize, recycle-invalid) are not separate
netdev_stat_ops or ethtool -S keys; each queue's NAPI increments
rx_qstats[].dropped and ndo_get_stats64() sums that into rtnl
rx_dropped; the poll paths no longer bump netdev->stats.rx_dropped.
TX large sends stay in the existing tx_large_packets key only, with
no per-queue key and no tx-hw-gso-packets: whether anything segments
a large send depends on where the frame goes.
TX hw_drops is left unset: yaml tx-hw-drops is packets that
arrived at the device, and dropped_packets also counts
software xmit rejects. ndo_get_stats64 still sums those
into tx_dropped, same as SQ.
.ndo_get_stats64() sums the per-queue packet and byte counters into
64-bit device totals. ethtool -S keeps the driver-specific keys with
no qstats field of their own: interrupts, polls, large_packets and
invalid_buffers per RX queue (rx-hw-drops folds invalid slots in with
the no-buffer count); send_failures per TX queue.
ETH_SS_STATS becomes variable-length because that block
scales with the live queue count. Hypercall counters and pool%d_ keys
are not added: which hcall a batch picks is not ABI, and size/active
already have sysfs.
The thirteen existing ethtool -S keys keep their exact names, their
order and their adapter-wide values, summed from the per-queue slots
on read. The storage moved; that ABI did not, except that
rx_no_buffer becomes cumulative across a reopen (below). The four
replenish_* counters get per-queue storage but no per-queue key of
their own.
Holding that ABI while the storage moves needs the ethtool -S table
to record where each key lives. IBMVETH_STAT_OFF() could only express
an offset into struct ibmveth_adapter. Tag every entry with an enum
ibmveth_stat_src naming the struct it indexes: adapter-wide keys are
read directly, per-queue keys are summed across the slots by one pair
of offset-keyed helpers. That is what lets the field names change
while the key names do not (rx_invalid_buffer now reads
invalid_buffers, tx_send_failed reads send_failures, and the two
large_packets fields live in different structs). tx_map_failed still
reads from the adapter -- it has no writer, here or in mainline --
and the three fw_enabled_* keys are capability flags, not counters.
The per-queue keys come from their own tables with the counts derived
by ARRAY_SIZE(), so get_strings(), get_ethtool_stats() and
get_sset_count() cannot drift apart.
ndo_get_stats64() walks every allocated slot rather than only the live
queues, so device totals cannot go backwards when ethtool -L shrinks
the queue count. get_base_stats() therefore reports the retired-queue
remainder rather than zero; the core sums it with the live queues it
iterates itself. Zeroing would assert that the live-queue sum is
already complete. Every field the per-queue callbacks fill is also
initialised there, because netdev_nl_stats_add() drops a field from
the device total unless both sides set it.
Give every queue a no_buffer_retired carry. PHYP's drop counter is
absolute for the buffer-list page currently mapped, so a reopen or a
queue reuse restarts it near zero. Storing only the newest absolute in
adapter->rx_no_buffer meant whichever queue ran last won, and the
value could go backwards. The carry sits beside the no_buffer_drops it
accumulates from, so both belong to one queue. Whenever
free_all_queues() lets the caller unmap, fold the last sample into
retired and clear drops while the page is still mapped; as close()
did before, that includes a single-queue H_FREE failure. Live
replenish still stores the page absolute; if the reading drops, the
previous absolute is folded into retired first. Readers snapshot the pair
under that queue's replenish_lock so a page-reset cannot be counted
twice. Every exported no_buffer value sums the two: the adapter-wide
rx_no_buffer key, rx_missed_errors, and per-queue rx-hw-drops and
rx-hw-drop-overruns, so none goes backwards across a close.
remove() frees the per-queue statistics arrays after
unregister_netdev() and after its H_FREE_LOGICAL_LAN retry, which
retires into them; the reset work is already disabled at the top of
remove().
ibmveth_probe_cleanup() also cancels the reset work and clears the
vio drvdata before free_netdev(), so drvdata does not outlive the
netdev on a probe failure, which never reaches ibmveth_remove().
Readers do not test the arrays for NULL: both exist from before
register_netdev() until after unregister_netdev(), with the reset
work already disabled, and probe fails -ENOMEM if either allocation
does.
Signed-off-by: Mingming Cao <[email protected]>
Reviewed-by: Dave Marquardt <[email protected]>
Tested-by: Shaik Abdulla <[email protected]>
---
Changes in v8:
- the per-queue no_buffer carry is folded by free_all_queues() before
every unmap, not only when a lower sample is seen, and read under
replenish_lock, so no exported value goes backwards across a close
- counters use WRITE_ONCE()/READ_ONCE() (IBMVETH_STAT_INC/ADD)
- ndo_get_stats64() fills rx_dropped and rx_missed_errors; rx_qstats
dropped counts bad-correlator, failed-harvest, recycle-invalid and
oversize slots
- qstats: rx-hw-drop-overruns filled; TX hw_drops unset; rx%d_polls
skips budget 0
- ethtool -S drops tx%d_checksum_offload (counter removed),
rx%d_no_buffer_drops (in qstats) and tx%d_large_packets (the
summed tx_large_packets stays)
- remove() frees the qstats arrays after its H_FREE retry
Changes in v7:
- close() harvests no_buffer under replenish_lock
- remove() uses put_pool_kobjs so the wait lands
before free_netdev
- remove-path unregister/cancel reorder and reset gate
moved to P10 with the reset producer
- noted: -S / get_stats64 / hw_drops mapping stays;
irqsave / free_buffer_pool / netpoll_poll_disable
stay leftover
Changes in v6:
- wrap the qstats local in replenish (81 cols)
- replenish_* per-queue u64, summed on the existing adapter-wide keys;
no per-queue replenish key; no atomics
- netdev_stat_ops for packets/bytes/drops, not private -S strings
- enum ibmveth_stat_src so existing -S keys keep names while storage
moves; ARRAY_SIZE() for the per-queue key counts
- no hcall_* keys (buffer-submit and H_SEND_LOGICAL_LAN)
- drop the three pool%d_ keys (15 ethtool entries)
- drop the fifteen qstats NULL checks outside the allocators
(five in the RX hot path)
- per-queue no_buffer_retired carry
- get_base_stats() reports the retired-queue remainder
- gate reset on NETREG_REGISTERED
- noted: harvest no_buffer on -L shrink is patch 14
Changes in v5:
- Series renumber: mailed v4 10/14 stats -> tip P11 (P09 peel;
get_channels -> P12)
- rx_no_buffer_retired + sum MAX_* slots so adapter no-buffer / qstat
totals stay monotonic across reopen and channel shrink
- probe_cleanup: clear vio drvdata before free_netdev (CMO cannot see a
freed netdev on rebind)
- remove: unregister_netdev then cancel_work_sync (no UAF reset worker)
Changes in v4:
- Merge v3's separate RX and TX stats commits into one patch.
- Introduce rx_queue_stats / tx_qstats / NUM macros here (first use).
- Allocate/free qstats at probe/remove instead of open/close.
- Report adapter-level ethtool strings by summing per-queue counters on
read; drop aggregate_* helpers.
- Sum global rx_no_buffer across MQ queues into this statistics patch.
- Cacheline-align per-queue stats; derive field counts with offsetof so
alignment padding is not counted as a statistic.
- probe_cleanup() cancels reset work, puts pool kobjects via helper from
the prior patch, and frees qstats on probe failure paths.
- Keep plain u64 qstats like existing ibmveth / ibmvnic (PPC_PSERIES).
drivers/net/ethernet/ibm/ibmveth.c | 549 ++++++++++++++++++++++++++---
drivers/net/ethernet/ibm/ibmveth.h | 60 +++-
2 files changed, 542 insertions(+), 67 deletions(-)
diff --git a/drivers/net/ethernet/ibm/ibmveth.c
b/drivers/net/ethernet/ibm/ibmveth.c
index a0c3c06d2b9d..45adf6e9be09 100644
--- a/drivers/net/ethernet/ibm/ibmveth.c
+++ b/drivers/net/ethernet/ibm/ibmveth.c
@@ -38,6 +38,7 @@
#include <asm/firmware.h>
#include <net/tcp.h>
#include <net/ip6_checksum.h>
+#include <net/netdev_queues.h>
#include "ibmveth.h"
@@ -75,32 +76,101 @@ module_param(old_large_send, bool, 0444);
MODULE_PARM_DESC(old_large_send,
"Use old large send method on firmware that supports the new method");
+/**
+ * enum ibmveth_stat_src - where an ethtool -S counter is stored
+ * @IBMVETH_STAT_ADAPTER: plain u64 in struct ibmveth_adapter
+ * @IBMVETH_STAT_RX_QSUM: per-queue u64, summed over rx_qstats[]
+ * @IBMVETH_STAT_TX_QSUM: per-queue u64, summed over tx_qstats[]
+ * @IBMVETH_STAT_RX_NO_BUFFER: rx_qstats[] live-page absolute plus the
+ * absolutes carried over from pages the queue has already retired
+ *
+ * Counters live per-queue so multi-queue writers never share a field.
+ * The adapter is only ever read from ethtool, so summing there is free.
+ */
+enum ibmveth_stat_src {
+ IBMVETH_STAT_ADAPTER,
+ IBMVETH_STAT_RX_QSUM,
+ IBMVETH_STAT_TX_QSUM,
+ IBMVETH_STAT_RX_NO_BUFFER,
+};
+
struct ibmveth_stat {
char name[ETH_GSTRING_LEN];
- int offset;
+ enum ibmveth_stat_src src;
+ /* Offset into the struct named by @src. */
+ size_t off;
};
#define IBMVETH_STAT_OFF(stat) offsetof(struct ibmveth_adapter, stat)
+#define IBMVETH_RXQ_OFF(stat) offsetof(struct ibmveth_rx_queue_stats, stat)
+#define IBMVETH_TXQ_OFF(stat) offsetof(struct ibmveth_tx_queue_stats, stat)
+
+/* Single writer per counter; lockless readers use READ_ONCE(). */
+#define IBMVETH_STAT_ADD(stat, n) WRITE_ONCE((stat), (stat) + (n))
+#define IBMVETH_STAT_INC(stat) IBMVETH_STAT_ADD(stat, 1)
#define IBMVETH_GET_STAT(a, off) *((u64 *)(((unsigned long)(a)) + off))
+#define IBMVETH_ADAPTER_STAT(key, field) \
+ { key, IBMVETH_STAT_ADAPTER, IBMVETH_STAT_OFF(field) }
+#define IBMVETH_RXQ_STAT(key, field) \
+ { key, IBMVETH_STAT_RX_QSUM, IBMVETH_RXQ_OFF(field) }
+#define IBMVETH_TXQ_STAT(key, field) \
+ { key, IBMVETH_STAT_TX_QSUM, IBMVETH_TXQ_OFF(field) }
+
+/*
+ * Key names and their order are ABI. Do not reorder or rename; append
+ * only, and only when the counter is worth a permanent interface.
+ */
static struct ibmveth_stat ibmveth_stats[] = {
- { "replenish_task_cycles", IBMVETH_STAT_OFF(replenish_task_cycles) },
- { "replenish_no_mem", IBMVETH_STAT_OFF(replenish_no_mem) },
- { "replenish_add_buff_failure",
- IBMVETH_STAT_OFF(replenish_add_buff_failure) },
- { "replenish_add_buff_success",
- IBMVETH_STAT_OFF(replenish_add_buff_success) },
- { "rx_invalid_buffer", IBMVETH_STAT_OFF(rx_invalid_buffer) },
- { "rx_no_buffer", IBMVETH_STAT_OFF(rx_no_buffer) },
- { "tx_map_failed", IBMVETH_STAT_OFF(tx_map_failed) },
- { "tx_send_failed", IBMVETH_STAT_OFF(tx_send_failed) },
- { "fw_enabled_ipv4_csum", IBMVETH_STAT_OFF(fw_ipv4_csum_support) },
- { "fw_enabled_ipv6_csum", IBMVETH_STAT_OFF(fw_ipv6_csum_support) },
- { "tx_large_packets", IBMVETH_STAT_OFF(tx_large_packets) },
- { "rx_large_packets", IBMVETH_STAT_OFF(rx_large_packets) },
- { "fw_enabled_large_send", IBMVETH_STAT_OFF(fw_large_send_support) }
+ IBMVETH_RXQ_STAT("replenish_task_cycles", replenish_task_cycles),
+ IBMVETH_RXQ_STAT("replenish_no_mem", replenish_no_mem),
+ IBMVETH_RXQ_STAT("replenish_add_buff_failure",
+ replenish_add_buff_failure),
+ IBMVETH_RXQ_STAT("replenish_add_buff_success",
+ replenish_add_buff_success),
+ IBMVETH_RXQ_STAT("rx_invalid_buffer", invalid_buffers),
+ { "rx_no_buffer", IBMVETH_STAT_RX_NO_BUFFER, 0 },
+ IBMVETH_ADAPTER_STAT("tx_map_failed", tx_map_failed),
+ IBMVETH_TXQ_STAT("tx_send_failed", send_failures),
+ IBMVETH_ADAPTER_STAT("fw_enabled_ipv4_csum", fw_ipv4_csum_support),
+ IBMVETH_ADAPTER_STAT("fw_enabled_ipv6_csum", fw_ipv6_csum_support),
+ IBMVETH_TXQ_STAT("tx_large_packets", large_packets),
+ IBMVETH_RXQ_STAT("rx_large_packets", large_packets),
+ IBMVETH_ADAPTER_STAT("fw_enabled_large_send", fw_large_send_support),
+};
+
+/**
+ * struct ibmveth_qstat - a per-queue counter exposed through ethtool -S
+ * @fmt: key name, taking the queue index as its only argument
+ * @off: offset into the matching per-queue stats struct
+ *
+ * Driving the strings and the values from one table keeps the two in
+ * step; get_sset_count() derives its length from ARRAY_SIZE() so the
+ * three cannot drift apart.
+ */
+struct ibmveth_qstat {
+ const char *fmt;
+ size_t off;
+};
+
+/*
+ * packets, bytes, drops and no-buffer overruns are reported through
+ * netdev_stat_ops. The keys below stay driver-specific.
+ */
+static const struct ibmveth_qstat ibmveth_rx_qstat_keys[] = {
+ { "rx%d_interrupts", IBMVETH_RXQ_OFF(interrupts) },
+ { "rx%d_polls", IBMVETH_RXQ_OFF(polls) },
+ { "rx%d_large_packets", IBMVETH_RXQ_OFF(large_packets) },
+ { "rx%d_invalid_buffers", IBMVETH_RXQ_OFF(invalid_buffers) },
};
+static const struct ibmveth_qstat ibmveth_tx_qstat_keys[] = {
+ { "tx%d_send_failures", IBMVETH_TXQ_OFF(send_failures) },
+};
+
+#define IBMVETH_NUM_RX_QSTATS ARRAY_SIZE(ibmveth_rx_qstat_keys)
+#define IBMVETH_NUM_TX_QSTATS ARRAY_SIZE(ibmveth_tx_qstat_keys)
+
/* simple methods of getting data from the current rxq entry */
static u32 ibmveth_rxq_flags(struct ibmveth_adapter *adapter,
int queue_index)
@@ -241,6 +311,60 @@ ibmveth_free_filter_list(struct ibmveth_adapter *adapter)
}
}
+/**
+ * ibmveth_alloc_rx_qstats - Allocate per-queue RX statistics
+ * @adapter: ibmveth adapter structure
+ *
+ * Return: 0 on success, -ENOMEM on failure
+ */
+static int ibmveth_alloc_rx_qstats(struct ibmveth_adapter *adapter)
+{
+ adapter->rx_qstats = kcalloc(IBMVETH_MAX_RX_QUEUES,
+ sizeof(*adapter->rx_qstats),
+ GFP_KERNEL);
+ if (!adapter->rx_qstats)
+ return -ENOMEM;
+
+ return 0;
+}
+
+/**
+ * ibmveth_free_rx_qstats - Free per-queue RX statistics
+ * @adapter: ibmveth adapter structure
+ */
+static void ibmveth_free_rx_qstats(struct ibmveth_adapter *adapter)
+{
+ kfree(adapter->rx_qstats);
+ adapter->rx_qstats = NULL;
+}
+
+/**
+ * ibmveth_alloc_tx_qstats - Allocate per-queue TX statistics
+ * @adapter: ibmveth adapter structure
+ *
+ * Return: 0 on success, -ENOMEM on failure
+ */
+static int ibmveth_alloc_tx_qstats(struct ibmveth_adapter *adapter)
+{
+ adapter->tx_qstats = kcalloc(IBMVETH_MAX_QUEUES,
+ sizeof(*adapter->tx_qstats),
+ GFP_KERNEL);
+ if (!adapter->tx_qstats)
+ return -ENOMEM;
+
+ return 0;
+}
+
+/**
+ * ibmveth_free_tx_qstats - Free per-queue TX statistics
+ * @adapter: ibmveth adapter structure
+ */
+static void ibmveth_free_tx_qstats(struct ibmveth_adapter *adapter)
+{
+ kfree(adapter->tx_qstats);
+ adapter->tx_qstats = NULL;
+}
+
/**
* ibmveth_alloc_rx_queues - Allocate per-queue RX resources
* @adapter: ibmveth adapter structure
@@ -861,6 +985,14 @@ struct ibmveth_replenish_fail {
u32 batch;
};
+static void ibmveth_count_add_fail(struct ibmveth_rx_queue_stats *qstats,
+ bool no_mem)
+{
+ if (no_mem)
+ IBMVETH_STAT_INC(qstats->replenish_no_mem);
+ IBMVETH_STAT_INC(qstats->replenish_add_buff_failure);
+}
+
/* Replenish the buffers for a pool. The skbs are not skb_reserve()d:
* PHYP reports the offset of each received frame.
* Caller must hold the per-queue replenish_lock.
@@ -873,6 +1005,8 @@ static int ibmveth_replenish_buffer_pool(struct
ibmveth_adapter *adapter,
int queue_index,
struct ibmveth_replenish_fail *fail)
{
+ struct ibmveth_rx_queue_stats *qstats =
+ &adapter->rx_qstats[queue_index];
union ibmveth_buf_desc descs[IBMVETH_MAX_RX_PER_HCALL] = {0};
u32 remaining = pool->size - atomic_read(&pool->available);
u64 correlators[IBMVETH_MAX_RX_PER_HCALL] = {0};
@@ -899,7 +1033,7 @@ static int ibmveth_replenish_buffer_pool(struct
ibmveth_adapter *adapter,
for (filled = 0; filled < min(remaining, batch); filled++) {
index = pool->free_map[free_index];
if (index == IBM_VETH_INVALID_MAP) {
- adapter->replenish_add_buff_failure++;
+ ibmveth_count_add_fail(qstats, false);
outcome = IBMVETH_REPLENISH_RESET_MAP;
break;
}
@@ -910,8 +1044,7 @@ static int ibmveth_replenish_buffer_pool(struct
ibmveth_adapter *adapter,
skb = netdev_alloc_skb(adapter->netdev,
pool->buff_size);
if (!skb) {
- adapter->replenish_no_mem++;
- adapter->replenish_add_buff_failure++;
+ ibmveth_count_add_fail(qstats, true);
break;
}
@@ -927,7 +1060,7 @@ static int ibmveth_replenish_buffer_pool(struct
ibmveth_adapter *adapter,
DMA_ATTR_NO_WARN);
if (dma_mapping_error(dev, dma_addr)) {
dev_kfree_skb_any(skb);
- adapter->replenish_add_buff_failure++;
+ ibmveth_count_add_fail(qstats, false);
break;
}
@@ -988,7 +1121,7 @@ static int ibmveth_replenish_buffer_pool(struct
ibmveth_adapter *adapter,
}
buffers_added += filled;
- adapter->replenish_add_buff_success += filled;
+ IBMVETH_STAT_ADD(qstats->replenish_add_buff_success, filled);
remaining -= filled;
memset(&descs, 0, sizeof(descs));
@@ -1011,7 +1144,7 @@ static int ibmveth_replenish_buffer_pool(struct
ibmveth_adapter *adapter,
pool->skbuff[index] = NULL;
}
}
- adapter->replenish_add_buff_failure += filled;
+ IBMVETH_STAT_ADD(qstats->replenish_add_buff_failure, filled);
if (lpar_rc == H_FUNCTION) {
if (adapter->multi_queue) {
@@ -1052,6 +1185,7 @@ static int ibmveth_replenish_buffer_pool(struct
ibmveth_adapter *adapter,
static void ibmveth_update_rx_no_buffer(struct ibmveth_adapter *adapter,
int queue_index)
{
+ struct ibmveth_rx_queue_stats *qstats;
__be64 *p;
u64 drops;
@@ -1063,7 +1197,45 @@ static void ibmveth_update_rx_no_buffer(struct
ibmveth_adapter *adapter,
p = adapter->buffer_list_addr[queue_index] + 4096 - 8;
drops = be64_to_cpup(p);
- adapter->rx_no_buffer = drops;
+ qstats = &adapter->rx_qstats[queue_index];
+
+ if (drops < qstats->no_buffer_drops)
+ IBMVETH_STAT_ADD(qstats->no_buffer_retired,
+ qstats->no_buffer_drops);
+ WRITE_ONCE(qstats->no_buffer_drops, drops);
+}
+
+/* Snapshot live+retired under replenish_lock; see retire_rx_no_buffer(). */
+static u64 ibmveth_rx_no_buffer_pair(struct ibmveth_adapter *adapter, int i)
+{
+ struct ibmveth_rx_q *rxq = &adapter->rx_queue[i];
+ struct ibmveth_rx_queue_stats *qstats = &adapter->rx_qstats[i];
+ u64 drops, retired;
+
+ spin_lock_bh(&rxq->replenish_lock);
+ drops = qstats->no_buffer_drops;
+ retired = qstats->no_buffer_retired;
+ spin_unlock_bh(&rxq->replenish_lock);
+ return drops + retired;
+}
+
+/*
+ * Call whenever the caller is about to unmap the buffer-list page.
+ * Take the last sample, then fold it into retired so the next page
+ * cannot overwrite the total.
+ */
+static void ibmveth_retire_rx_no_buffer(struct ibmveth_adapter *adapter,
+ int queue_index)
+{
+ struct ibmveth_rx_q *rxq = &adapter->rx_queue[queue_index];
+ struct ibmveth_rx_queue_stats *qstats;
+
+ spin_lock_bh(&rxq->replenish_lock);
+ ibmveth_update_rx_no_buffer(adapter, queue_index);
+ qstats = &adapter->rx_qstats[queue_index];
+ IBMVETH_STAT_ADD(qstats->no_buffer_retired, qstats->no_buffer_drops);
+ WRITE_ONCE(qstats->no_buffer_drops, 0);
+ spin_unlock_bh(&rxq->replenish_lock);
}
/* replenish routine */
@@ -1088,7 +1260,7 @@ static void ibmveth_replenish_task(struct ibmveth_adapter
*adapter,
spin_lock_bh(&rxq->replenish_lock);
- adapter->replenish_task_cycles++;
+ IBMVETH_STAT_INC(adapter->rx_qstats[queue_index].replenish_task_cycles);
for (i = (IBMVETH_NUM_BUFF_POOLS - 1); i >= 0; i--) {
struct ibmveth_buff_pool *pool =
@@ -1875,6 +2047,9 @@ static void ibmveth_release_lan_resources(struct
ibmveth_adapter *adapter)
* reason it cannot simply keep the memory mapped is that on a CMO
* partition vio_cmo_bus_remove() BUG()s on an outstanding entitlement.
*
+ * Whenever the caller will unmap, retire each queue's no_buffer
+ * sample first, while the page is still mapped.
+ *
* On return 0, clears registered flags and cookies of queues
* [0, num_rx_queues); queue_irq[] is released by
* ibmveth_cleanup_rx_interrupts() on close, or by
@@ -1900,6 +2075,9 @@ static int ibmveth_free_all_queues(struct ibmveth_adapter
*adapter)
return -EIO;
}
+ for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++)
+ ibmveth_retire_rx_no_buffer(adapter, i);
+
for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++)
ibmveth_clear_rx_queue_handle(adapter, i);
@@ -2172,7 +2350,6 @@ static int ibmveth_open(struct net_device *netdev)
static int ibmveth_close(struct net_device *netdev)
{
struct ibmveth_adapter *adapter = netdev_priv(netdev);
- int i;
/* change_mtu, pool sysfs, set_csum and set_tso call close() and
* open() directly. If that open() fails, IFF_UP stays set and
@@ -2189,8 +2366,6 @@ static int ibmveth_close(struct net_device *netdev)
ibmveth_cleanup_rx_interrupts(adapter);
if (!ibmveth_free_all_queues(adapter)) {
- for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++)
- ibmveth_update_rx_no_buffer(adapter, i);
/* netif_tx_disable() above keeps xmit off the TX LTBs. */
ibmveth_release_lan_resources(adapter);
}
@@ -2453,22 +2628,100 @@ static int ibmveth_set_features(struct net_device *dev,
return rc1 ? rc1 : rc2;
}
-static void ibmveth_get_strings(struct net_device *dev, u32 stringset, u8
*data)
+/*
+ * Sum per-queue counters for rare ethtool reads. The hot paths only ever
+ * touch their own queue's slot, so nothing here needs an atomic; the cost
+ * of aggregation is paid by the reader instead (ibmvnic-style).
+ *
+ * Every slot is summed, not just the live ones, so that shrinking the
+ * queue count with ethtool -L cannot make a counter go backwards.
+ */
+static u64 ibmveth_sum_rx_qstat(struct ibmveth_adapter *adapter, size_t off)
{
+ u64 total = 0;
int i;
+ for (i = 0; i < IBMVETH_MAX_RX_QUEUES; i++) {
+ const u8 *q = (const u8 *)&adapter->rx_qstats[i];
+
+ total += READ_ONCE(*(const u64 *)(q + off));
+ }
+
+ return total;
+}
+
+static u64 ibmveth_sum_tx_qstat(struct ibmveth_adapter *adapter, size_t off)
+{
+ u64 total = 0;
+ int i;
+
+ for (i = 0; i < IBMVETH_MAX_QUEUES; i++) {
+ const u8 *q = (const u8 *)&adapter->tx_qstats[i];
+
+ total += READ_ONCE(*(const u64 *)(q + off));
+ }
+
+ return total;
+}
+
+static u64 ibmveth_ethtool_adapter_stat(struct ibmveth_adapter *adapter,
+ int index)
+{
+ const struct ibmveth_stat *stat = &ibmveth_stats[index];
+
+ switch (stat->src) {
+ case IBMVETH_STAT_RX_QSUM:
+ return ibmveth_sum_rx_qstat(adapter, stat->off);
+ case IBMVETH_STAT_TX_QSUM:
+ return ibmveth_sum_tx_qstat(adapter, stat->off);
+ case IBMVETH_STAT_RX_NO_BUFFER: {
+ u64 total = 0;
+ int i;
+
+ for (i = 0; i < IBMVETH_MAX_RX_QUEUES; i++)
+ total += ibmveth_rx_no_buffer_pair(adapter, i);
+ return total;
+ }
+ case IBMVETH_STAT_ADAPTER:
+ break;
+ }
+
+ return IBMVETH_GET_STAT(adapter, stat->off);
+}
+
+static void ibmveth_get_strings(struct net_device *dev, u32 stringset, u8
*data)
+{
+ struct ibmveth_adapter *adapter = netdev_priv(dev);
+ u8 *p = data;
+ int i, j;
+
if (stringset != ETH_SS_STATS)
return;
- for (i = 0; i < ARRAY_SIZE(ibmveth_stats); i++, data += ETH_GSTRING_LEN)
- memcpy(data, ibmveth_stats[i].name, ETH_GSTRING_LEN);
+ for (i = 0; i < ARRAY_SIZE(ibmveth_stats); i++) {
+ memcpy(p, ibmveth_stats[i].name, ETH_GSTRING_LEN);
+ p += ETH_GSTRING_LEN;
+ }
+
+ for (i = 0; i < ibmveth_get_num_rx_queues(adapter); i++)
+ for (j = 0; j < IBMVETH_NUM_RX_QSTATS; j++)
+ ethtool_sprintf(&p, ibmveth_rx_qstat_keys[j].fmt, i);
+
+ for (i = 0; i < dev->real_num_tx_queues; i++)
+ for (j = 0; j < IBMVETH_NUM_TX_QSTATS; j++)
+ ethtool_sprintf(&p, ibmveth_tx_qstat_keys[j].fmt, i);
}
static int ibmveth_get_sset_count(struct net_device *dev, int sset)
{
+ struct ibmveth_adapter *adapter = netdev_priv(dev);
+
switch (sset) {
case ETH_SS_STATS:
- return ARRAY_SIZE(ibmveth_stats);
+ return ARRAY_SIZE(ibmveth_stats) +
+ ibmveth_get_num_rx_queues(adapter) *
+ IBMVETH_NUM_RX_QSTATS +
+ dev->real_num_tx_queues * IBMVETH_NUM_TX_QSTATS;
default:
return -EOPNOTSUPP;
}
@@ -2477,11 +2730,27 @@ static int ibmveth_get_sset_count(struct net_device
*dev, int sset)
static void ibmveth_get_ethtool_stats(struct net_device *dev,
struct ethtool_stats *stats, u64 *data)
{
- int i;
struct ibmveth_adapter *adapter = netdev_priv(dev);
+ int i, j, k;
for (i = 0; i < ARRAY_SIZE(ibmveth_stats); i++)
- data[i] = IBMVETH_GET_STAT(adapter, ibmveth_stats[i].offset);
+ data[i] = ibmveth_ethtool_adapter_stat(adapter, i);
+
+ for (j = 0; j < ibmveth_get_num_rx_queues(adapter); j++) {
+ const u8 *q = (const u8 *)&adapter->rx_qstats[j];
+
+ for (k = 0; k < IBMVETH_NUM_RX_QSTATS; k++)
+ data[i++] = READ_ONCE(*(const u64 *)
+ (q + ibmveth_rx_qstat_keys[k].off));
+ }
+
+ for (j = 0; j < dev->real_num_tx_queues; j++) {
+ const u8 *q = (const u8 *)&adapter->tx_qstats[j];
+
+ for (k = 0; k < IBMVETH_NUM_TX_QSTATS; k++)
+ data[i++] = READ_ONCE(*(const u64 *)
+ (q + ibmveth_tx_qstat_keys[k].off));
+ }
}
static void ibmveth_get_channels(struct net_device *netdev,
@@ -2605,8 +2874,10 @@ static int ibmveth_send(struct ibmveth_adapter *adapter,
}
static int ibmveth_is_packet_unsupported(struct sk_buff *skb,
- struct net_device *netdev)
+ struct ibmveth_adapter *adapter,
+ int queue_num)
{
+ struct net_device *netdev = adapter->netdev;
struct ethhdr *ether_header;
int ret = 0;
@@ -2614,7 +2885,7 @@ static int ibmveth_is_packet_unsupported(struct sk_buff
*skb,
if (ether_addr_equal(ether_header->h_dest, netdev->dev_addr)) {
netdev_dbg(netdev, "veth doesn't support loopback packets,
dropping packet.\n");
- netdev->stats.tx_dropped++;
+ IBMVETH_STAT_INC(adapter->tx_qstats[queue_num].dropped_packets);
ret = -EOPNOTSUPP;
}
@@ -2625,20 +2896,23 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff
*skb,
struct net_device *netdev)
{
struct ibmveth_adapter *adapter = netdev_priv(netdev);
+ struct ibmveth_tx_queue_stats *txs;
unsigned int desc_flags, total_bytes;
union ibmveth_buf_desc desc;
int i, queue_num = skb_get_queue_mapping(skb);
unsigned long mss = 0;
+ txs = &adapter->tx_qstats[queue_num];
+
/* TX LTBs are freed on shrink; a sender that picked its queue
* before the shrink can get here after the free.
*/
if (unlikely(!adapter->tx_ltb_ptr[queue_num])) {
- netdev->stats.tx_dropped++;
+ IBMVETH_STAT_INC(txs->dropped_packets);
goto out;
}
- if (ibmveth_is_packet_unsupported(skb, netdev))
+ if (ibmveth_is_packet_unsupported(skb, adapter, queue_num))
goto out;
/* veth can't checksum offload UDP */
if (skb->ip_summed == CHECKSUM_PARTIAL &&
@@ -2649,7 +2923,7 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,
skb_checksum_help(skb)) {
netdev_err(netdev, "tx: failed to checksum packet\n");
- netdev->stats.tx_dropped++;
+ IBMVETH_STAT_INC(txs->dropped_packets);
goto out;
}
@@ -2672,7 +2946,7 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,
if (skb->ip_summed == CHECKSUM_PARTIAL && skb_is_gso(skb)) {
if (adapter->fw_large_send_support) {
mss = (unsigned long)skb_shinfo(skb)->gso_size;
- adapter->tx_large_packets++;
+ IBMVETH_STAT_INC(txs->large_packets);
} else if (!skb_is_gso_v6(skb)) {
/* Put -1 in the IP checksum to tell phyp it
* is a largesend packet. Put the mss in
@@ -2681,7 +2955,7 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,
ip_hdr(skb)->check = 0xffff;
tcp_hdr(skb)->check =
cpu_to_be16(skb_shinfo(skb)->gso_size);
- adapter->tx_large_packets++;
+ IBMVETH_STAT_INC(txs->large_packets);
}
}
@@ -2689,7 +2963,7 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,
if (unlikely(skb->len > adapter->tx_ltb_size)) {
netdev_err(adapter->netdev, "tx: packet size (%u) exceeds ltb
(%u)\n",
skb->len, adapter->tx_ltb_size);
- netdev->stats.tx_dropped++;
+ IBMVETH_STAT_INC(txs->dropped_packets);
goto out;
}
memcpy(adapter->tx_ltb_ptr[queue_num], skb->data, skb_headlen(skb));
@@ -2706,7 +2980,7 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff *skb,
if (unlikely(total_bytes != skb->len)) {
netdev_err(adapter->netdev, "tx: incorrect packet len copied
into ltb (%u != %u)\n",
skb->len, total_bytes);
- netdev->stats.tx_dropped++;
+ IBMVETH_STAT_INC(txs->dropped_packets);
goto out;
}
desc.fields.flags_len = desc_flags | skb->len;
@@ -2715,11 +2989,11 @@ static netdev_tx_t ibmveth_start_xmit(struct sk_buff
*skb,
dma_wmb();
if (ibmveth_send(adapter, desc.desc, mss)) {
- adapter->tx_send_failed++;
- netdev->stats.tx_dropped++;
+ IBMVETH_STAT_INC(txs->send_failures);
+ IBMVETH_STAT_INC(txs->dropped_packets);
} else {
- netdev->stats.tx_packets++;
- netdev->stats.tx_bytes += skb->len;
+ IBMVETH_STAT_INC(txs->packets);
+ IBMVETH_STAT_ADD(txs->bytes, skb->len);
}
out:
@@ -2887,7 +3161,7 @@ static void ibmveth_rx_csum_helper(struct sk_buff *skb,
static void ibmveth_poll_bump_invalid(struct ibmveth_adapter *adapter,
int queue_index)
{
- adapter->rx_invalid_buffer++;
+ IBMVETH_STAT_INC(adapter->rx_qstats[queue_index].invalid_buffers);
}
static bool ibmveth_poll_stopping(struct net_device *netdev,
@@ -2909,7 +3183,7 @@ static bool ibmveth_poll_recycle_invalid(struct
net_device *netdev,
rc = ibmveth_rxq_harvest_correlator(adapter, queue_index, true,
correlator);
if (unlikely(rc))
- netdev->stats.rx_dropped++;
+ IBMVETH_STAT_INC(adapter->rx_qstats[queue_index].dropped);
return rc;
}
@@ -2922,7 +3196,7 @@ static void ibmveth_poll_skip_bad_correlator(struct
net_device *netdev,
* the harvest logs the correlator when it fails.
*/
schedule_work(&adapter->work);
- netdev->stats.rx_dropped++;
+ IBMVETH_STAT_INC(adapter->rx_qstats[queue_index].dropped);
ibmveth_rxq_harvest_correlator(adapter, queue_index, true, correlator);
}
@@ -2936,7 +3210,7 @@ static void ibmveth_poll_drop_oversize(struct net_device
*netdev,
netdev_err(netdev,
"RX frame %u+%u exceeds buffer %u on queue %d,
dropping\n",
off, len, room, queue_index);
- netdev->stats.rx_dropped++;
+ IBMVETH_STAT_INC(adapter->rx_qstats[queue_index].dropped);
ibmveth_rxq_harvest_correlator(adapter, queue_index, true, correlator);
}
@@ -2955,6 +3229,7 @@ static int ibmveth_poll_deliver_frame(struct napi_struct
*napi,
struct net_device *netdev,
int queue_index)
{
+ struct ibmveth_rx_queue_stats *rxs = &adapter->rx_qstats[queue_index];
struct sk_buff *skb, *new_skb;
unsigned int room, off, len, pool;
struct ibmveth_buff_pool *bpool;
@@ -3013,7 +3288,7 @@ static int ibmveth_poll_deliver_frame(struct napi_struct
*napi,
correlator);
if (unlikely(rc)) {
dev_kfree_skb_any(new_skb);
- netdev->stats.rx_dropped++;
+ IBMVETH_STAT_INC(rxs->dropped);
return -1;
}
skb = new_skb;
@@ -3021,7 +3296,7 @@ static int ibmveth_poll_deliver_frame(struct napi_struct
*napi,
rc = ibmveth_rxq_harvest_correlator(adapter, queue_index,
false, correlator);
if (unlikely(rc)) {
- netdev->stats.rx_dropped++;
+ IBMVETH_STAT_INC(rxs->dropped);
return -1;
}
skb_reserve(skb, offset);
@@ -3044,7 +3319,7 @@ static int ibmveth_poll_deliver_frame(struct napi_struct
*napi,
iph_check == 0xffff) {
if (!ibmveth_rx_mss_helper(skb, mss, lrg_pkt))
csum_good = 0;
- adapter->rx_large_packets++;
+ IBMVETH_STAT_INC(rxs->large_packets);
}
if (csum_good) {
@@ -3055,8 +3330,8 @@ static int ibmveth_poll_deliver_frame(struct napi_struct
*napi,
skb_record_rx_queue(skb, queue_index);
napi_gro_receive(napi, skb);
- netdev->stats.rx_packets++;
- netdev->stats.rx_bytes += length;
+ IBMVETH_STAT_INC(rxs->packets);
+ IBMVETH_STAT_ADD(rxs->bytes, length);
return 1;
}
@@ -3083,6 +3358,10 @@ static int ibmveth_poll(struct napi_struct *napi, int
budget)
return 0;
}
+ /* budget 0 is netpoll; polls stays single-writer. */
+ if (budget)
+ IBMVETH_STAT_INC(adapter->rx_qstats[queue_index].polls);
+
while (frames_processed < budget) {
if (ibmveth_poll_stopping(netdev, napi))
break;
@@ -3178,6 +3457,8 @@ static irqreturn_t ibmveth_interrupt(int irq, void
*dev_instance)
if (qindex < 0 || qindex >= ibmveth_get_num_rx_queues(adapter))
return IRQ_NONE;
+ IBMVETH_STAT_INC(adapter->rx_qstats[qindex].interrupts);
+
ibmveth_schedule_rx_queue(adapter, qindex);
return IRQ_HANDLED;
}
@@ -3383,6 +3664,133 @@ static netdev_features_t ibmveth_features_check(struct
sk_buff *skb,
return vlan_features_check(skb, features);
}
+/**
+ * ibmveth_get_stats64 - Return aggregated per-queue statistics
+ * @dev: network device
+ * @stats: rtnl link statistics storage
+ *
+ * Sums per-queue rx_qstats and tx_qstats into the rtnl counters.
+ * rx_missed_errors is the PHYP no-buffer pair (live + retired),
+ * which if_link.h counts separately from rx_dropped.
+ * Walk the full allocated arrays (not the live queue count) so shrinking
+ * channels cannot make the totals go backwards.
+ */
+static void ibmveth_get_stats64(struct net_device *dev,
+ struct rtnl_link_stats64 *stats)
+{
+ struct ibmveth_adapter *adapter = netdev_priv(dev);
+ int i;
+
+ for (i = 0; i < IBMVETH_MAX_RX_QUEUES; i++) {
+ stats->rx_packets += READ_ONCE(adapter->rx_qstats[i].packets);
+ stats->rx_bytes += READ_ONCE(adapter->rx_qstats[i].bytes);
+ stats->rx_dropped += READ_ONCE(adapter->rx_qstats[i].dropped);
+ stats->rx_missed_errors +=
+ ibmveth_rx_no_buffer_pair(adapter, i);
+ }
+
+ for (i = 0; i < IBMVETH_MAX_QUEUES; i++) {
+ stats->tx_packets += READ_ONCE(adapter->tx_qstats[i].packets);
+ stats->tx_bytes += READ_ONCE(adapter->tx_qstats[i].bytes);
+ stats->tx_dropped +=
+ READ_ONCE(adapter->tx_qstats[i].dropped_packets);
+ }
+}
+
+static void ibmveth_get_queue_stats_rx(struct net_device *dev, int idx,
+ struct netdev_queue_stats_rx *stats)
+{
+ struct ibmveth_adapter *adapter = netdev_priv(dev);
+ u64 no_buffer;
+
+ stats->packets = READ_ONCE(adapter->rx_qstats[idx].packets);
+ stats->bytes = READ_ONCE(adapter->rx_qstats[idx].bytes);
+ /*
+ * Only frames PHYP itself dropped: the no-buffer pair (live page
+ * plus pages this queue already released) and slots PHYP handed
+ * back marked invalid. Frames the driver rejects count in
+ * rx_dropped instead.
+ */
+ no_buffer = ibmveth_rx_no_buffer_pair(adapter, idx);
+ stats->hw_drops = no_buffer +
+ READ_ONCE(adapter->rx_qstats[idx].invalid_buffers);
+ stats->hw_drop_overruns = no_buffer;
+ stats->alloc_fail = READ_ONCE(adapter->rx_qstats[idx].replenish_no_mem);
+}
+
+static void ibmveth_get_queue_stats_tx(struct net_device *dev, int idx,
+ struct netdev_queue_stats_tx *stats)
+{
+ struct ibmveth_adapter *adapter = netdev_priv(dev);
+
+ stats->packets = READ_ONCE(adapter->tx_qstats[idx].packets);
+ stats->bytes = READ_ONCE(adapter->tx_qstats[idx].bytes);
+ /* TX hw_drops left unset: dropped_packets includes
+ * software xmit rejects that never reached PHYP.
+ */
+}
+
+/**
+ * ibmveth_get_base_stats - account for traffic not on a live queue
+ * @dev: network device
+ * @rx: RX base statistics storage
+ * @tx: TX base statistics storage
+ *
+ * get_queue_stats_{rx,tx}() only report queues the core still iterates,
+ * i.e. below real_num_{rx,tx}_queues, while ibmveth_get_stats64() walks
+ * the full arrays so device totals stay monotonic across a shrink.
+ * Report the retired-queue remainder here, otherwise qstats and
+ * rtnl_link_stats64 disagree by a delta that grows with every shrink.
+ * Zeroing would not be neutral: per netdev_stat_ops it asserts the
+ * per-queue sum is already exact.
+ *
+ * Bound the live side with real_num_*_queues rather than the adapter's
+ * own count, so the split lines up with the core's iteration exactly.
+ *
+ * Every field the per-queue callbacks fill must also be initialised
+ * here: netdev_nl_stats_add() starts the sum at NETDEV_STAT_NOT_SET and
+ * only accumulates while both sides are set, so a field left unset here
+ * is dropped from the device total even though the queues report it.
+ */
+static void ibmveth_get_base_stats(struct net_device *dev,
+ struct netdev_queue_stats_rx *rx,
+ struct netdev_queue_stats_tx *tx)
+{
+ struct ibmveth_adapter *adapter = netdev_priv(dev);
+ unsigned int i;
+
+ rx->packets = 0;
+ rx->bytes = 0;
+ rx->alloc_fail = 0;
+ rx->hw_drops = 0;
+ rx->hw_drop_overruns = 0;
+ tx->packets = 0;
+ tx->bytes = 0;
+
+ for (i = dev->real_num_rx_queues; i < IBMVETH_MAX_RX_QUEUES; i++) {
+ u64 no_buffer = ibmveth_rx_no_buffer_pair(adapter, i);
+
+ rx->packets += READ_ONCE(adapter->rx_qstats[i].packets);
+ rx->bytes += READ_ONCE(adapter->rx_qstats[i].bytes);
+ rx->hw_drops += no_buffer +
+ READ_ONCE(adapter->rx_qstats[i].invalid_buffers);
+ rx->hw_drop_overruns += no_buffer;
+ rx->alloc_fail +=
+ READ_ONCE(adapter->rx_qstats[i].replenish_no_mem);
+ }
+
+ for (i = dev->real_num_tx_queues; i < IBMVETH_MAX_QUEUES; i++) {
+ tx->packets += READ_ONCE(adapter->tx_qstats[i].packets);
+ tx->bytes += READ_ONCE(adapter->tx_qstats[i].bytes);
+ }
+}
+
+static const struct netdev_stat_ops ibmveth_stat_ops = {
+ .get_queue_stats_rx = ibmveth_get_queue_stats_rx,
+ .get_queue_stats_tx = ibmveth_get_queue_stats_tx,
+ .get_base_stats = ibmveth_get_base_stats,
+};
+
static const struct net_device_ops ibmveth_netdev_ops = {
.ndo_open = ibmveth_open,
.ndo_stop = ibmveth_close,
@@ -3395,6 +3803,7 @@ static const struct net_device_ops ibmveth_netdev_ops = {
.ndo_validate_addr = eth_validate_addr,
.ndo_set_mac_address = ibmveth_set_mac_addr,
.ndo_features_check = ibmveth_features_check,
+ .ndo_get_stats64 = ibmveth_get_stats64,
};
/**
@@ -3433,6 +3842,23 @@ static void ibmveth_put_pool_kobjs(struct
ibmveth_adapter *adapter,
wait_for_completion(&adapter->rx_buff_pool[0][i].released);
}
+static void ibmveth_probe_cleanup(struct ibmveth_adapter *adapter,
+ int pools_ready)
+{
+ struct net_device *netdev = adapter->netdev;
+
+ cancel_work_sync(&adapter->work);
+ ibmveth_put_pool_kobjs(adapter, pools_ready);
+
+ ibmveth_free_tx_qstats(adapter);
+ ibmveth_free_rx_qstats(adapter);
+ /* Probe failure never reaches ibmveth_remove(); clear before free
+ * so drvdata does not outlive the netdev.
+ */
+ dev_set_drvdata(&adapter->vdev->dev, NULL);
+ free_netdev(netdev);
+}
+
static int ibmveth_probe(struct vio_dev *dev, const struct vio_device_id *id)
{
int rc, i, mac_len, pools_ready = 0;
@@ -3497,9 +3923,16 @@ static int ibmveth_probe(struct vio_dev *dev, const
struct vio_device_id *id)
netif_napi_add_weight(netdev, &adapter->napi[i],
ibmveth_poll, 16);
+ if (ibmveth_alloc_rx_qstats(adapter) ||
+ ibmveth_alloc_tx_qstats(adapter)) {
+ ibmveth_probe_cleanup(adapter, 0);
+ return -ENOMEM;
+ }
+
netdev->irq = dev->irq;
netdev->netdev_ops = &ibmveth_netdev_ops;
netdev->ethtool_ops = &netdev_ethtool_ops;
+ netdev->stat_ops = &ibmveth_stat_ops;
SET_NETDEV_DEV(netdev, &dev->dev);
netdev->hw_features = NETIF_F_SG;
if (vio_get_attribute(dev, "ibm,illan-options", NULL) != NULL) {
@@ -3634,8 +4067,7 @@ static int ibmveth_probe(struct vio_dev *dev, const
struct vio_device_id *id)
return 0;
err_put_pools:
- ibmveth_put_pool_kobjs(adapter, pools_ready);
- free_netdev(netdev);
+ ibmveth_probe_cleanup(adapter, pools_ready);
return rc;
}
@@ -3658,6 +4090,9 @@ static void ibmveth_remove(struct vio_dev *dev)
ibmveth_release_lan_resources(adapter);
}
+ ibmveth_free_tx_qstats(adapter);
+ ibmveth_free_rx_qstats(adapter);
+
free_netdev(netdev);
dev_set_drvdata(&dev->dev, NULL);
}
diff --git a/drivers/net/ethernet/ibm/ibmveth.h
b/drivers/net/ethernet/ibm/ibmveth.h
index 393d37add10c..76640bd5598e 100644
--- a/drivers/net/ethernet/ibm/ibmveth.h
+++ b/drivers/net/ethernet/ibm/ibmveth.h
@@ -283,6 +283,46 @@ static int pool_active[] = { 1, 1, 0, 0, 1};
#define IBM_VETH_INVALID_MAP ((u16)0xffff)
+/*
+ * Per-queue RX counters. No field has two concurrent writers:
+ * interrupts is written only from this queue's IRQ handler; polls,
+ * packets, bytes, large_packets, invalid_buffers and dropped only from
+ * its NAPI poll (polls not from netpoll's budget-0 call);
+ * replenish_* only under its replenish_lock; and no_buffer_drops and
+ * no_buffer_retired only under that lock. Readers snapshot that pair
+ * under the same lock. No atomic or u64_stats_sync is therefore
+ * needed: the driver is PPC64-only, so 64-bit loads and stores do not
+ * tear. Readers still run locklessly on other CPUs, so writers use
+ * IBMVETH_STAT_INC()/IBMVETH_STAT_ADD() and readers READ_ONCE(), which
+ * keeps the compiler from splitting or refetching the accesses.
+ */
+struct ibmveth_rx_queue_stats {
+ u64 packets;
+ u64 bytes;
+ u64 interrupts;
+ u64 polls;
+ u64 large_packets;
+ u64 invalid_buffers;
+ u64 dropped;
+ /* PHYP's per-page absolute drop count for the live page. */
+ u64 no_buffer_drops;
+ /* Absolutes from pages this queue has already retired. */
+ u64 no_buffer_retired;
+ u64 replenish_task_cycles;
+ u64 replenish_no_mem;
+ u64 replenish_add_buff_failure;
+ u64 replenish_add_buff_success;
+} ____cacheline_aligned_in_smp;
+
+/* Per-queue TX counters; serialized by the stack's per-queue TX lock. */
+struct ibmveth_tx_queue_stats {
+ u64 packets;
+ u64 bytes;
+ u64 large_packets;
+ u64 dropped_packets;
+ u64 send_failures;
+} ____cacheline_aligned_in_smp;
+
struct ibmveth_buff_pool {
u32 size;
u32 index;
@@ -347,17 +387,17 @@ struct ibmveth_adapter {
u64 fw_ipv6_csum_support;
u64 fw_ipv4_csum_support;
u64 fw_large_send_support;
- /* adapter specific stats */
- u64 replenish_task_cycles;
- u64 replenish_no_mem;
- u64 replenish_add_buff_failure;
- u64 replenish_add_buff_success;
- u64 rx_invalid_buffer;
- u64 rx_no_buffer;
+ /*
+ * Every other ethtool -S counter lives in rx_qstats/tx_qstats and is
+ * summed on read. tx_map_failed predates multi-queue, has never been
+ * updated by any code path, and is kept only so the key keeps
+ * reporting the zero userspace already sees.
+ */
u64 tx_map_failed;
- u64 tx_send_failed;
- u64 tx_large_packets;
- u64 rx_large_packets;
+
+ struct ibmveth_rx_queue_stats *rx_qstats;
+ struct ibmveth_tx_queue_stats *tx_qstats;
+
/* Ethtool settings */
u8 duplex;
u32 speed;
--
2.50.1 (Apple Git-155)
