date:20200526

Re: [dpdk-dev] [dpdk-techboard] Consider improving the DPDKcontribution processes

2020-05-26 Thread Tom Barbette


Le 25/05/2020 à 22:34, Thomas Monjalon a écrit :

25/05/2020 20:44, Morten Brørup:

From: Thomas Monjalon

25/05/2020 18:09, Burakov, Anatoly:

obviously, but i have a suspicion that we'll get more of it if we

lower

the barrier for entry (not the barrier for merge!). I think there is

a

way to lower the secondary skill level needed to contribute to DPDK
without lowering coding/merge standards with it.


That is exactly what I am asking for: Lowering the barrier and increasing the 
feeling of success for newcomers. (The barrier for merge is probably fine; I'll 
leave that discussion to the maintainers.)


I understand.



About the barrier for entry, maybe it is not obvious because I don't
communicate a lot about it, but please be aware that I (and other
maintainers I think) are doing a lot of changes in newcomer patches
to avoid asking them knowing the whole process from the beginning.
Then frequent contributors get educated on the way.


Great! I wish that every developer would think and behave this way.



I think the only real barrier we have is to sign the patch
with a real name and send an email to right list.
The ask for SoB real name is probably what started this thread
in Morten's mind. And the SoB requirement will *never* change.


The incorrect Signed-off-by might be the only hard barrier (which we cannot 
avoid). But that did not trigger me.

I was raising the discussion to bring attention to soft barriers for 
contributors. What triggered me was the request to split the patch into 
multiple patches; a kind of feedback I have seen before. For an experienced git 
user, this is probably very easy, but for a git newbie (like myself), it 
basically means starting all over and trying to figure out the right set of git 
commands to do this, which can be perceived as a difficult task requiring a lot 
of effort.


Yes I am aware about this difficulty.
It is basically knowing git-reset and git-add -p.
I agree a cookbook for this kind of thing is required.

I would like to do the split for newcomers,
but we need also to validate the explanations of each commit.
A solution in such case is to send the split so the newbie can just
fill what is missing.
This kind of workflow is really what we should look at improving.



Perhaps we could supplement the Contributor Guidelines with a set of cookbooks 
for different steps in the contribution process, so reviewers can be refer 
newcomers to the relevant of these as part of the feedback. Just like any 
professional customer support team has a set of canned answers ready for common 
customer issues. (Please note: I am not suggesting adding an AI/ML chat bot 
reviewer to the mailing list!)


OK



The amount of Contributor Guideline documentation is also a balance... it must 
be long enough to contain the relevant information to get going, but short 
enough for newcomers to bother reading it.


Yes, we need short intros and long explanations when really needed.
It is touching another issue: we lack some documentation love.





Maybe we could find something that allows to "git push" to the 
patchwork, where it kind of appears already as a github-like discussion? 
 It doesn't miss a lot to enable writing from the website directly 
(basically auto-email).


Personnaly I've put a lot of efforts to fix simple comments, be sure 
that I wrote "v2" here, sign-off there, cc-ed the right person, not mess 
my the format-patch versions, changed only the cover letter, ... Quite 
afraid of bothering that big mailing list for nothing.


I'm infrequent enough to have te re-learn every time basically. It would 
be much easier with a git push, a fast online review of the diff, as on 
github/gitlab, and done. Also, those allow online edits, and therefore 
allows "elders" to do small fixes directly in the "patch". Some fixes 
are not worth the discussion and the chain of mails. That's what I'm 
missing the most personnaly.


Tom

Re: [dpdk-dev] [dpdk-stable] [PATCH v2] test-pmd: fix memory leaks when mtr policer actions update fails

2020-05-26 Thread Thomas Monjalon

26/05/2020 04:04, wangyunjian:
> From: Thomas Monjalon [mailto:tho...@monjalon.net]
> > 25/05/2020 03:46, wangyunjian:
> > > From: Yunjian Wang 
> > >
> > > This patch fixes the Huawei internal coverity reported resource leak
> > > issue.
> > 
> > The problem is not seen in the community Coverity?
> 
> I don't know much about that.

Please, could you register and check here?
https://scan.coverity.com/projects/dpdk-data-plane-development-kit

Re: [dpdk-dev] [dpdk-techboard] Consider improving the DPDKcontribution processes

2020-05-26 Thread Maxime Coquelin




On 5/26/20 9:06 AM, Tom Barbette wrote:
> Le 25/05/2020 à 22:34, Thomas Monjalon a écrit :
>> 25/05/2020 20:44, Morten Brørup:
>>> From: Thomas Monjalon
 25/05/2020 18:09, Burakov, Anatoly:
> obviously, but i have a suspicion that we'll get more of it if we
 lower
> the barrier for entry (not the barrier for merge!). I think there is
 a
> way to lower the secondary skill level needed to contribute to DPDK
> without lowering coding/merge standards with it.
>>>
>>> That is exactly what I am asking for: Lowering the barrier and
>>> increasing the feeling of success for newcomers. (The barrier for
>>> merge is probably fine; I'll leave that discussion to the maintainers.)
>>
>> I understand.
>>
>>
 About the barrier for entry, maybe it is not obvious because I don't
 communicate a lot about it, but please be aware that I (and other
 maintainers I think) are doing a lot of changes in newcomer patches
 to avoid asking them knowing the whole process from the beginning.
 Then frequent contributors get educated on the way.
>>>
>>> Great! I wish that every developer would think and behave this way.
>>>

 I think the only real barrier we have is to sign the patch
 with a real name and send an email to right list.
 The ask for SoB real name is probably what started this thread
 in Morten's mind. And the SoB requirement will *never* change.
>>>
>>> The incorrect Signed-off-by might be the only hard barrier (which we
>>> cannot avoid). But that did not trigger me.
>>>
>>> I was raising the discussion to bring attention to soft barriers for
>>> contributors. What triggered me was the request to split the patch
>>> into multiple patches; a kind of feedback I have seen before. For an
>>> experienced git user, this is probably very easy, but for a git
>>> newbie (like myself), it basically means starting all over and trying
>>> to figure out the right set of git commands to do this, which can be
>>> perceived as a difficult task requiring a lot of effort.
>>
>> Yes I am aware about this difficulty.
>> It is basically knowing git-reset and git-add -p.
>> I agree a cookbook for this kind of thing is required.
>>
>> I would like to do the split for newcomers,
>> but we need also to validate the explanations of each commit.
>> A solution in such case is to send the split so the newbie can just
>> fill what is missing.
>> This kind of workflow is really what we should look at improving.
>>
>>
>>> Perhaps we could supplement the Contributor Guidelines with a set of
>>> cookbooks for different steps in the contribution process, so
>>> reviewers can be refer newcomers to the relevant of these as part of
>>> the feedback. Just like any professional customer support team has a
>>> set of canned answers ready for common customer issues. (Please note:
>>> I am not suggesting adding an AI/ML chat bot reviewer to the mailing
>>> list!)
>>
>> OK
>>
>>
>>> The amount of Contributor Guideline documentation is also a
>>> balance... it must be long enough to contain the relevant information
>>> to get going, but short enough for newcomers to bother reading it.
>>
>> Yes, we need short intros and long explanations when really needed.
>> It is touching another issue: we lack some documentation love.
>>
>>
> 
> 
> Maybe we could find something that allows to "git push" to the
> patchwork, where it kind of appears already as a github-like discussion?
>  It doesn't miss a lot to enable writing from the website directly
> (basically auto-email).
> 
> Personnaly I've put a lot of efforts to fix simple comments, be sure
> that I wrote "v2" here, sign-off there, cc-ed the right person, not mess
> my the format-patch versions, changed only the cover letter, ... Quite
> afraid of bothering that big mailing list for nothing.

Maybe using git-publish would help here:
https://github.com/stefanha/git-publish

Using the simple git-puslish command, it manages revisions
automatically, open an editor for the cover letter, can run some scripts
to add proper maintainers, and hook available to run basic checks,
etc...

We could add a .gitpublish file to automate adding right maintainers
depending on the branch, etc...

For example, for Qemu the .gitpublish file looks like this:
https://github.com/qemu/qemu/blob/master/.gitpublish

> I'm infrequent enough to have te re-learn every time basically. It would
> be much easier with a git push, a fast online review of the diff, as on
> github/gitlab, and done. Also, those allow online edits, and therefore
> allows "elders" to do small fixes directly in the "patch". Some fixes
> are not worth the discussion and the chain of mails. That's what I'm
> missing the most personnaly.
> 
> Tom
>

Re: [dpdk-dev] [PATCH] mbuf: document rule for new fields and flags

2020-05-26 Thread Jerin Jacob

On Tue, May 26, 2020 at 2:54 AM Thomas Monjalon  wrote:
>
> Since dynamic fields and flags were added in 19.11,
> the idea was to use them for new features, not only PMD-specific.
>
> The rule is made more explicit in doxygen, in the mbuf guide,
> and in the contribution design guidelines.
>
> For more information about the original design, see the presentation
> https://www.dpdk.org/wp-content/uploads/sites/35/2019/10/DynamicMbuf.pdf
>
> Signed-off-by: Thomas Monjalon 
> ---
>  doc/guides/contributing/design.rst | 13 +
>  doc/guides/prog_guide/mbuf_lib.rst | 23 +++
>  lib/librte_mbuf/rte_mbuf_core.h|  2 ++
>  3 files changed, 38 insertions(+)
>
> diff --git a/doc/guides/contributing/design.rst 
> b/doc/guides/contributing/design.rst
> index d3dd694b65..508115d5bd 100644
> --- a/doc/guides/contributing/design.rst
> +++ b/doc/guides/contributing/design.rst
> @@ -57,6 +57,19 @@ The following config options can be used:
>  * ``CONFIG_RTE_EXEC_ENV`` is a string that contains the name of the 
> executive environment.
>  * ``CONFIG_RTE_EXEC_ENV_FREEBSD`` or ``CONFIG_RTE_EXEC_ENV_LINUX`` are 
> defined only if we are building for this execution environment.
>
> +Mbuf features
> +-
> +
> +The ``rte_mbuf`` structure must be kept small (128 bytes).
> +
> +In order to add new features without wasting buffer space for unused 
> features,
> +some fields and flags can be registered dynamically in a shared area.

I think, instead of "can", it should be "must"

> +The "dynamic" mbuf area is the default choice for the new features.
> +
> +The "dynamic" area is eating the remaining space in mbuf,
> +and some existing "static" fields may need to become "dynamic".
> +
> +
>  Library Statistics
>  --
>
> diff --git a/doc/guides/prog_guide/mbuf_lib.rst 
> b/doc/guides/prog_guide/mbuf_lib.rst
> index 0d3223b081..c3dbfb9221 100644
> --- a/doc/guides/prog_guide/mbuf_lib.rst
> +++ b/doc/guides/prog_guide/mbuf_lib.rst
> @@ -207,6 +207,29 @@ The list of flags and their precise meaning is described 
> in the mbuf API
>  documentation (rte_mbuf.h). Also refer to the testpmd source code
>  (specifically the csumonly.c file) for details.
>
> +Dynamic fields and flags
> +
> +
> +The size of the mbuf is constrained and limited;
> +while the amount of metadata to save for each packet is quite unlimited.
> +The most basic networking information already find their place
> +in the existing mbuf fields and flags.
> +
> +If new features need to be added, the new fields and flags should fit

How about, instead of "should fit", must use the dynamic space.

> +in the "dynamic space", by registering some room in the mbuf structure:
> +
> +dynamic field
> +   named area in the mbuf structure,
> +   with a given size (at least 1 byte) and alignment constraint.
> +
> +dynamic flag
> +   named bit in the mbuf structure,
> +   stored in the field ``ol_flags``.
> +
> +The dynamic fields and flags are managed with the functions 
> ``rte_mbuf_dyn*``.
> +
> +It is not possible to unregister fields or flags.
> +
>  .. _direct_indirect_buffer:
>
>  Direct and Indirect Buffers
> diff --git a/lib/librte_mbuf/rte_mbuf_core.h b/lib/librte_mbuf/rte_mbuf_core.h
> index b9a59c879c..22be41e520 100644
> --- a/lib/librte_mbuf/rte_mbuf_core.h
> +++ b/lib/librte_mbuf/rte_mbuf_core.h
> @@ -12,6 +12,8 @@
>   * packet offload flags and some related macros.
>   * For majority of DPDK entities, it is not recommended to include
>   * this file directly, use include  instead.
> + *
> + * New fields and flags should fit in the "dynamic space".

must use.

>   */
>
>  #include 
> --
> 2.26.2
>

Re: [dpdk-dev] mlx5 & pdump: convert HW timestamps to nanoseconds

2020-05-26 Thread Tom Barbette

Le 22/05/2020 à 20:43, PATRICK KEROULAS a écrit :

mlx5 part of libibverbs includes a ts-to-ns converter which takes the
instantaneous clock info. It's unused in dpdk so far. I've tested it in the
device/port init routine and the result looks reliable. Since this approach
looks very simple, compared to the time sync mechanism, I'm trying to
integrate.

The conversion should occur in the primary process (testpmd) I suppose.
1) The needed clock info derives from ethernet device. Is it possible to
access that struct from a rx callback?
2) how to attach the nanosecond to mbuf so that `pdump` catches it?
(workaround: copy `mbuf->udata64` in forwarded packets.)
3) any other idea?

The timestamp is carried in mbuf.
Then the conversion must be done by the ethdev caller (application or
any other upper layer).

What if the converter function needs a clock_info?
https://github.com/linux-rdma/rdma-core/blob/7af01c79e00555207dee6132d72e7bfc1bb5485e/providers/mlx5/mlx5dv.h#L1201
I'm affraid this info may change by the time the converter is called
by upper layer.

Indeed, the clock in the device is not an atomic one :)
We need to adjust the time conversion continuously.
I am not an expert of time synchronization, so I add more people Cc
who could help for having a precise timestamp.

Thanks Thomas.
Not sure this is a synchronization issue. We have dedicated processes
(linuxptp) to keep both NIC and sys clocks in sync with an external clock.
It is "just" a matter of unit conversion.

If it has to be performed in dpdk-pdump, I would need some help to
retrieve mlx5_clock_info from inside a secondary process. Looking at
mlx5_read_clock(), this info is extracted from ibv_context which looks
reachable in a primary process only (segfault, if I try in pdump).

I don't know about the integrated ts-to-ns, but we implemented a
translation mechanism that mimics what NTP does in Linux to translate a
given clock (TSC at first) to a wall time. You'll find more info at
https://orbi.uliege.be/bitstream/2268/226257/1/thesis.pdf chapter
3.4.1. This is an often forgotten matter, as we saw in real switches
that the time spent in time-related VDSO is enormous.

We wanted to do a very precise capture too, se we made that clock able
to synchronize itself with the ConnectX 5 internal clock as a base
instead of TSC. FYI the clock in CX5 si running at 800MHz, so pure
nanosecond is impossible, but close enough. It is for that purpose that
I proposed the rte_eth_read_clock() patch in DPDK. We need to be able to
read the current clock (like rdtsc() instruction for TSC) to compute the
frequency.

The "converter" code is there :
https://github.com/tbarbette/fastclick/blob/master/elements/userlevel/tscclock.cc,
the source is configurable (TSC, rte_eth_read_clock, GPS meinberg clock,
...), the DPDK one is there :
https://github.com/tbarbette/fastclick/blob/2ab021283b82d0b980551480c505ec8dced98e0a/elements/userlevel/dpdkdevclock.cc#L27

One important thing is that the conversion factor must be changed from
time to time to fix the drifiting. That is the reason why we can't just
push a bunch of code to DPDK (and it's probably not as simple as using
the ts-to-ns in mlx5) because you must have a timer, and use a RCU to
update "atomically" a > 64bits struct. Though most of that is available
now in DPDK but there will always be some setup (rcu barrier, timer
init, ...).

60 matches

Mail list logo