On 11/9/2018 2:27 PM, Thomas Monjalon wrote: > 09/11/2018 14:17, Thomas Monjalon: >> 09/11/2018 11:03, Ferruh Yigit: >>> On 11/8/2018 11:01 PM, Yongseok Koh wrote: >>>> >>>>> On Nov 8, 2018, at 9:21 AM, Ferruh Yigit <ferruh.yi...@intel.com> wrote: >>>>> >>>>> On 11/8/2018 3:59 PM, Thomas Monjalon wrote: >>>>>> Hi, >>>>>> >>>>>> We need to gather more information about this bug. >>>>>> More below. >>>>>> >>>>>> 07/11/2018 10:04, Wiles, Keith: >>>>>>>> On Nov 6, 2018, at 9:30 PM, Yongseok Koh <ys...@mellanox.com> wrote: >>>>>>>>> On Nov 5, 2018, at 6:06 AM, Wiles, Keith <keith.wi...@intel.com> >>>>>>>>> wrote: >>>>>>>>>> On Nov 2, 2018, at 9:04 PM, Yongseok Koh <ys...@mellanox.com> wrote: >>>>>>>>>> >>>>>>>>>> This is a workaround to prevent a crash, which might be caused by >>>>>>>>>> optimization of newer gcc (7.3.0) on Intel Skylake. >>>>>>>>> >>>>>>>>> Should the code below not also test for the gcc version and >>>>>>>>> the Sky Lake processor, maybe I am wrong but it seems it is >>>>>>>>> turning AVX512 for all GCC builds >>>>>>>> >>>>>>>> I didn't want to check gcc version as 7.3.0 is very new. Only gcc 8 is >>>>>>>> newly up since then (gcc 8.2). >>>>>>>> Also, I wasn't able to test every gcc versions and I wanted to be a >>>>>>>> bit conservative for this crash. >>>>>>>> Performance drop (if any) by disabling a new (experimental) feature >>>>>>>> would be less risky than unaccountable crash. >>>>>>>> And, it does disable the feature only if CONFIG_RTE_ENABLE_AVX512=n. >>>>>>>> Please refer to v3. >>>>>>> >>>>>>> Are you not turning off all of the GCC versions for AVX512. >>>>>>> And you can test for range or greater then GCC version and >>>>>>> it just seems like we are turning off every gcc version, is that true? >>>>>> >>>>>> Do we know exactly which GCC versions are affected? >>>>>> >>>>>>>>> Also bug 97 seems a bit obscure reference, maybe you know >>>>>>>>> the bug report, but more details would be good? >>>>>>>> >>>>>>>> I sent out the report to dev list two month ago. >>>>>>>> And I created the Bug 97 in order to reference it >>>>>>>> in the commit message. >>>>>>>> I didn't want to repeat same message here and there, >>>>>>>> but it would've been better to have some sort of summary >>>>>>>> of the Bug, although v3 has a few more words. >>>>>>>> However, v3 has been merged. >>>>>>> >>>>>>> Still this is too obscure if nothing else give a link to >>>>>>> a specific bug not just 97. >>>>>> >>>>>> The URL is >>>>>> >>>>>> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbugs.dpdk.org%2Fshow_bug.cgi%3Fid%3D97&data=02%7C01%7Cyskoh%40mellanox.com%7C90ff6c361faf422b976108d6459eb490%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636772945282345908&sdata=2o%2Fg203aWrKCYg16S6oI4BcS41igpLu1DloS%2FrRnknc%3D&reserved=0 >>>>>> The bug is also pointing to an email: >>>>>> >>>>>> https://emea01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmails.dpdk.org%2Farchives%2Fdev%2F2018-September%2F111522.html&data=02%7C01%7Cyskoh%40mellanox.com%7C90ff6c361faf422b976108d6459eb490%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C636772945282345908&sdata=NCFKxaREd69iZ8eyFKg%2FWBP73CLTXkxrNQQeii%2Bbsao%3D&reserved=0 >>>>>> >>>>>> Summary: >>>>>> - CPU: Intel Skylake >>>>>> - Linux environment: Ubuntu 18.04 >>>>>> - Compiler: gcc-7.3 (Ubuntu 7.3.0-16ubuntu3) >>>>> >>>>> Is it possible to test a few other gcc versions to check if the issue is >>>>> specific to this compiler version? >>>> >>>> Nothing's impossible but even with my quick search in gcc.gnu.org, >>>> I could find the following documents mention mavx512f support: >>>> >>>> GCC 4.9.0 >>>> April 22, 2014 (changes, documentation) >>>> >>>> GCC 5.1 >>>> April 22, 2015 (changes, documentation) >>>> >>>> GCC 6.4 >>>> July 4, 2017 (changes, documentation) >>>> >>>> GCC 7.1 >>>> May 2, 2017 (changes, documentation) >>>> >>>> GCC 8.1 >>>> May 2, 2018 (changes, documentation) >>>> >>>> We altogether have to put quite large resource to verify all of the >>>> versions. >>>> >>>> I assumed older than gcc 7 would have the same issue. I know it was a >>>> speculation >>>> but like I mentioned I wanted to be more conservative. I didn't mean this >>>> is a permanent fix. >>>> For two months, we couldn't have any tangible solution (actually nobody >>>> cared including myself), >>>> so I submitted the patch to temporarily disable mavx512f. >>>> >>>> I'm still not sure what the best option is... >>> >>> For permanent fix we need more information, currently we can't re-produce >>> this >>> defect. Since you can reproduce it we need your support. >>> >>> Right now we don't know if this is compiler issue or code defect in >>> rte_memcpy() >>> or something else. >>> >>> It is easy to disable mavx512f as temporarily solution but it is coming >>> with the >>> cost of the performance drop, also without knowing the actual root cause I >>> wouldn't say this is being conservative, actual issue may be just hidden >>> with >>> this change. >>> >>> I think as first thing we need to find a way to reproduce this issue in any >>> other way than using mlx5 PMD. So that we can put more organized effort to >>> fix this. >>> I attached a simple unit test for rte_memcpy(), if this is a rte_memcpy() >>> with >>> avx512f defect as claimed, you should be able to see the issue with that, >>> right? >>> Did you able to find a chance to test it? Do you observer any crash there? >> >> I am able to connect to a machine where the issue is reproduced. >> So I have tested replacing rte_memcpy with memcpy, >> and the crash disappears when using memcpy. >> So it confirms that the issue is in rte_memcpy. > > One workaround is to disable CONFIG_RTE_ENABLE_AVX, > but it is disabling AVX and AVX2 for all DPDK code. > > A more limited fix (tested) can be to disable AVX2 version of rte_memcpy > and rely on the AVX version (which is not crashing): > > --- a/lib/librte_eal/common/include/arch/x86/rte_memcpy.h > +++ b/lib/librte_eal/common/include/arch/x86/rte_memcpy.h > -#elif defined RTE_MACHINE_CPUFLAG_AVX2 > +#elif defined RTE_MACHINE_CPUFLAG_AVX2_disable
I put a patch into bugzilla: https://bugs.dpdk.org/attachment.cgi?id=18&action=diff Can you please check if this workaround prevents the crash without performance drop. Also there is another suggestion from Yongseok, that looks simpler, but not covering CONFIG_RTE_ENABLE_AVX512=y case.
