Hi Oliver, >- optimize some code to avoid a real memory barrier when not required >(timers, virtio, ...)
That seems like a good thing to me. > - make the code more readable to distinguish between the 2 kinds of memory > barrier. That part seems a bit misleading to me. rte_compiler_barier() - is a barrier just for a compiler, not for real cpu. It only guarantees that the compiler wouldn't reorder instructions across it while emitting the code. Looking at Intel Memory Ordering rules (Intel System PG, section 8.2): 1) Reads may be reordered with older writes to different locations but not with older writes to the same location. So with the following fragment of code: int a; extern int *x, *y; L0: *y = 0; rte_compiler_barrier(); L1: a = *x; There is no guarantee that store at L0 will always be finished before load at L1. Which means to me that rte_smp_mb() can't be identical to compiler_barrier, but should be real 'mfence' instruction instead. 2) Writes to memory are not reordered with other writes, with the following exceptions: ... streaming stores (writes) executed with the non-temporal move instructions (MOVNTI, MOVNTQ, MOVNTDQ, MOVNTPS, and MOVNTPD); ... So with the following fragment of code: extern int *x; extern __128i a, *p; L0: _mm_stream_si128( p, a); rte_compiler_barrier(); L1: *x = 0; There is no guarantee that store at L0 will always be finished before store at L1. Which means to me that rte_smp_wmb() can't be identical to compiler_barrier, but should be real 'sfence' instruction instead. The only replacement that seems safe to me is: #define rte_smp_rmb() rte_compiler_barrier() But now, there seems a confusion: everyone has to remember that smp_mb() and smp_wmb() are 'real' fences, while smp_rmb() is not. That's why my suggestion was to simply keep using compiler_barrier() for all cases, when we don't need real fence. Thanks Konstantin -----Original Message----- From: Olivier MATZ [mailto:olivier.m...@6wind.com] Sent: Tuesday, May 20, 2014 1:13 PM To: Ananyev, Konstantin; dev at dpdk.org Subject: Re: [dpdk-dev] [PATCH] atomic: clarify use of memory barriers Hi Konstantin, Thank you for your review and feedback. On 05/20/2014 12:05 PM, Ananyev, Konstantin wrote: >> Note that on x86 CPUs, memory barriers between different cores can be >> guaranteed by a simple compiler barrier. > > I don't think this is totally correct. > Yes, for Intel cpus in many cases memory barrier could be avoided due to > nearly strict memory ordering. > Though there are few cases where reordering is possible and when fence > instructions would be needed. I tried to mimic the behavior of linux that differentiates *mb() from smp_*mb(), but I did too fast. In linux, we have [1]: smp_mb() = mb() = asm volatile("mfence":::"memory") smp_rmb() = compiler_barrier() smp_wmb() = compiler_barrier() At least this should fixed in the patch. By the way, just for reference, the idea of the patch came from a discussion we had on the list [2]. > For me: > +#define rte_smp_rmb() rte_compiler_barrier() > Seems a bit misleading, as there is no real fence. > So I suggest we keep rte_compiler_barrier() naming and usage. The objectives of the patch (which was probably not explained very clearly in the commit log) were: - make the code more readable to distinguish between the 2 kinds of memory barrier. - optimize some code to avoid a real memory barrier when not required (timers, virtio, ...) Having a compiler barrier in place of a memory barrier in the code does not really help to understand what the developper wanted to do. In the current code we can see that the use of rte_compiler_barrier() is ambiguous, as it need a comment to clarify the situation: rte_compiler_barrier(); /* rmb */ Don't you think we could fix the patch but keep its logic? Regards, Olivier [1] http://lxr.free-electrons.com/source/arch/x86/include/asm/barrier.h#L81 [2] http://dpdk.org/ml/archives/dev/2014-March/001741.html
