Hi Ori, Alexander,
On Tue, 1 Feb 2022, Ori Kam wrote:
Hi
-----Original Message-----
From: Ivan Malov <ivan.ma...@oktetlabs.ru>
Sent: Tuesday, February 1, 2022 2:18 AM
Subject: RE: [PATCH v2 03/10] ethdev: bring in async queue-based flow rules
Hi all,
On Thu, 27 Jan 2022, Alexander Kozyrev wrote:
On Wednesday, January 26, 2022 13:54 Ajit Khaparde <ajit.khapa...@broadcom.com>
wrote:
On Tue, Jan 25, 2022 at 9:03 PM Alexander Kozyrev <akozy...@nvidia.com>
wrote:
On Monday, January 24, 2022 19:00 Ivan Malov <ivan.ma...@oktetlabs.ru>
wrote:
This series is very helpful as it draws attention to
the problem of making flow API efficient. That said,
there is much room for improvement, especially in
what comes to keeping things clear and concise.
In example, the following APIs
- rte_flow_q_flow_create()
- rte_flow_q_flow_destroy()
- rte_flow_q_action_handle_create()
- rte_flow_q_action_handle_destroy()
- rte_flow_q_action_handle_update()
should probably be transformed into a unified one
int
rte_flow_q_submit_task(uint16_t port_id,
uint32_t queue_id,
const struct rte_flow_q_ops_attr *q_ops_attr,
enum rte_flow_q_task_type task_type,
const void *task_data,
rte_flow_q_task_cookie_t cookie,
struct rte_flow_error *error);
with a handful of corresponding enum defines and data structures
for these 5 operations.
We were thinking about the unified function for all queue operations.
Good.
But it has too many drawbacks in our opinion:
Is that so?
1. Different operation return different results and unneeded parameters.
q_flow_create gives a flow handle, q_action_handle returns indirect action
handle.
destroy functions return the status. All these cases needs to be handled
differently.
Yes, all of these are to be handled differently, but this does not mean
that one cannot think of a unified handle format. The application can
remember which cookie corresponds to which operation type after all.
I agree with with Alexander, merging all of those functions to one will result
in
extra parameters which will be unsued.
It is not clear which parameters are going to be unused in which cases.
Task type? Always used. Task data? Always used. Cookie? Presumably,
always used. But, as I said, you can either change the name to
your liking or prefer a task data-internal parameter to pass
task handles back to the caller instead of the "cookie".
Basic programing apporch is that functions should do one thing,
it is easier to document / use / test and will run faster.
By the sound of it, we are starting to run into disagreement in what comes
to being "easier to document". A unified API by its very definition should
be easier to document, extend and maintain.
However, I see your point. This design spares no expense to make
the code super-fast. I have no strong opinion on whether it is
good to abandon other basic programming points like keeping
things generic / unified and extensible, which are as
important as the point of having simpler functions.
Perhaps caring about performance indeed prevails here.
And maybe I should quit being that sceptical.
Also, the unified function is bloated with various parameters not needed
for all operations.
That is far-fetched.
Just thinking about extra parameter to return the handle only in case
of create flow.
One can make the task data argument "in-out" and keep the handle field
over there, union-style. And no handle for tasks which do not need one.
Also in future I guess we will want to add modify rule and indirect actions
I assume that in both case dedicated functions will be much better
for example for the indirect use the same names we are using now
only adding q in the name and extra queue parameter.
Non-unified set of APIs is also bloated. Takes long to read. Many
duplicating comments. When one has added a new API for a different
type of task, they will have to duplicate many lines one more time.
In the case of unified API, one has to add a new enum type (one line),
specific (and thus concise) description for it, and the corresponding
structure for the task data. That's it. The rest is up to the PMDs.
Also, it should be possible to make the task data IN-OUT, to return
its task-specific handle (to address your above concern).
Both of these point results in hard to understand API and messy
documentation with
various structures on how to use it in every particular case.
I disagree with you I think that having unused and misleading parameters
is much more messy from user view point.
Yes, but currently it is not clear which parameters
one deems unused or misleading and in which cases.
The documentation for the method is always the same. Precise and concise.
The task data structures will have their own descriptions, yes. But that
does not make the documentation messy. Or am I missing something?
2. Performance consideration.
We aimed the new API with the insertion/deletion rate in mind.
Good.
Adding if conditions to distinguish between requested operation will cause
some degradation.
Some degradation.. - how serious would it be? What's for the "if"
conditions, well, I believe the compiler is smart enough to deal
with them efficiently. After all, the suggested approach is
a fixed set of operation (task) types. One can have a
static array of task-specific methods in the PMD.
And only one condition to check the value bounds.
yes from our testing just an if will result in degradation.
Even just the indirect ref to the PMD function results in degradation.
I guess if you try to add an if in the datapath (rx_burst/tx_burst) you will
Need to explain the reason.
I see your point. But, in my understanding, this new design, first of all,
does not belong in *traffic* path (albeit it is meant for *fast path*,
in general), so it is not quite correct to compare it with Rx burst
functionality. Secondly, the idea is to make things non-blocking
and avoid tedious parsing (hence all the templates and tables),
and that goal is achieved in any case. So, removing *all* "if"
checks at any cost might not be that reasonable.
BTW, keeping things robust is another basic programming point...
It is preferred to have separate small functions that do one job and make it
efficient.
A noble idea.
Noble idea is good so lets do it small functions.
Well, I have no strong objections. You can try and see how it goes.
Interfaces are still the same.
That is the major point of confusion. The application developer has to
be super-careful to tell the queue version of "flow_create" from the
regular one. The two set of APIs are indeed counterparts, and that's
might be ambiguous. Whilst in the unified approach, readers will
understand that this new API is just a task-submitter for
the asynchronous type of operation.
Glad I made it clearer. Ivan, what do you think about these considerations?
Well, I'm not pushing anyone to abandon the existing approach and switch
to the unified API design. But the above points might not be that
difficult to address. This deserves more discussions.
Any other opinions?
I vote for the suggested API lets try and see.
Agreed. Sometimes it pays to try and see the results.
This API has experimental status anyway...
By the way, shouldn't this variety of operation types cover such
from the tunnel offload model? Getting PMD's opaque "tunnel
match" items and "tunnel set" actions - things like that.
Don't quite get the idea. Could you please elaborate more on this?
rte_flow_tunnel_decap_set(), rte_flow_tunnel_action_decap_release();
rte_flow_tunnel_match(), rte_flow_tunnel_item_release().
Also, I suggest that the attribute "drain"
be replaced by "postpone" (invert the meaning).
rte_flow_q_drain() should then be renamed to
rte_flow_q_push_postponed().
The rationale behind my suggestion is that "drain" tricks readers into
thinking that the enqueued operations are going to be completely
purged,
whilst the true intention of the API is to "push" them to the hardware.
I don't have a strong opinion on this naming, if you think "postpone" is
better.
Or we can name it as "doorbell" to signal a NIC about some work to be
done
and "rte_flow_q_doorbell" to do this explicitly after a few operations.
By the sound of it, "push" is just a shorter term for "doorbell".
I don't think the name should be doorbell, since it may not just be doorbell
for example PMD may organize the rules in a better way.
rte_flow_q_dequeue() also needs to be revisited. The name suggests
that
some "tasks" be cancelled, whereas in reality this API implies "poll"
semantics. So why not name it "rte_flow_q_poll"?
The polling implies an active busy-waiting of the result. Which is not the
case here.
That is far-fetched. Polling implies some periodic invocations,
that is correct. But it's up to the application whether to
make this a tight loop or not. Or am I missing something?
I like the dequeue since it also tells the developer that this is how he
remove rules from the queue.
Look. The major point here is that, according to the description,
task completions may be reordered. Hence, technically, this is
not a queue, not a ring, but rather a "postponed task list".
That being said, I believe that the term "queue" (like "q"
in the API names) can be retained - it's fine to call a
completion list a "queue". But in what comes to the
other ring-specific terms like "dequeue", I would
go for some other alternatives like "poll".
Please correct me if I've got that wrong.
What we do is only getting results for already processed operations, hence
"dequeue" as opposite to "queue".
What do you think? Or we can have push for drain and pull for dequeue as
an alternative.
Pull is not that informative. The commonly-used term is "poll".
But let's hear some other opinions. Maybe someone has their
way with naming.
I believe this function should return an array of completions, just
like it does in the current version, but provide a "cookie" (can be
represented by a uintptr_t value) for each completion entry.
The rationale behind choosing "cookie" over "user_data" is clarity.
Term "user_data" sounds like "flow action conf" or "counter data",
whilst in reality it likely stands for something normally called
a cookie. Please correct me if I've got that wrong.
I haven't heard about cookies in context not related to web browsing.
I think that user data is more than a simple cookie, it can contain
anything that application wants to associate with this flow rule, i.e.
connection ID, timestamp, anything. It is more general term here.
OK. Seems like I overestimated clarity of the term "cookie".
Well, the short of it: submit_task(), push_postponed() and poll().
Having a unified "submit_task()" API should allow to reduce the
amount of comment duplication.
I'm afraid it won't reduce the amount of comments needed anyway.
It will reduce the amount of generic (now just repeating) comments.
That's it. Task-specific comments will stay, and that is fine.
Like above API should be as direct as possible.
OK. Your point is very clear. I do not object.
But maybe more opinions exist regarding that.
Instead of 5 descriptions of purpose-specific function we will end up with
a huge blob of text trying to explain how to use a single function with
5 different input structures and 3 different output types. That is really
messy.
Is that so? It shouldn't be like that. It should say about enqueuing a
task, whilst task-specific comments belong in the corresponding enum
or structure definitions. Definitely not a single huge comment.
In what comes to RST commentary, please consider a bullet-formatted
description for improved readability:
- Flow rule management is done via special flow management queues
- The queue operations are asynchronous and not thread-safe
- The operations can thus be invoked by the app's datapath
- The queue count is configured at initialisation stage
- Available operation types: submit_task, push_postponed and poll
- Polling provides completions for submitted tasks
- The completions can be reordered withing a queue
- Polling must be done on time to avoid overflows
Agree, it does seem nicer and cleaner, will adopt this style in v3.
Very well.
Please note that the above review notes are just a quick primer,
nothing more. I may be mistaken with regard to some aspects.
I humbly request that we agree on the API sketch and naming
before going to the next review cycle.
Thank you.
Thanks for your suggestions, let's see if we can find a common round here.
Alright.
--
Ivan M
Best,
Ori
