Just a quick note to say that I've closed the survey. I haven't published the results yet as I said that I would write notes necessary as a preamble to correctly inform potential readers, and these notes are taking longer to write than I have time available.
Cheers, Etienne On Wed, Feb 24, 2021 at 7:07 PM Etienne-Victor Depasquale <ed...@ieee.org> wrote: > I think I need to calm this thread down. > > I'm a researcher, and my interest is in the truth, not in my opinion. > > I've read some facts in this thread that are necessary > as a prerequisite to the publication of the results on Friday. > > I do want to ensure that no future reader is misinformed and will do my > best, > with the help of contribution from my peers in this good community, > to summarize all objections to this survey's questions, > in the same message as that which publishes the result. > > All peace and good wishes, > > Etienne > > On Wed, Feb 24, 2021 at 4:35 PM Robert Bays <rob...@gdk.org> wrote: > >> To the nanog community, I’m sorry to have dragged this conversation out >> further. I'm only responding to this because there are a significant >> number of open source projects and commercial products that use DPDK, or >> similar userspace network environment in their implementations. The >> statements in this thread incorrectly cast them, because they use DPDK, as >> inefficient. But the reality is they have all been designed from day one >> not to unnecessarily consume power. Please ask your open source dev and/or >> vendor of choice to verify. But please don’t rely on the information in >> this thread to make decisions about what you deploy in your network. >> >> On Feb 23, 2021, at 11:44 PM, Etienne-Victor Depasquale <ed...@ieee.org> >> wrote: >> >> Hello Robert, >> >> Your statement that DPDK “keeps utilization at 100% regardless of packet >>> activity” is just not correct. You further pre-suppose "widespread DPDK's >>> core operating inefficiency” without any data to backup the operating >>> inefficacy assertion. >>> >> >> This statement is incorrect. >> I have provided references (please see earlier e-mails) that investigate >> the operation of DPDK. >> These references are items of peer-reviewed research that investigate a >> perceived problem with deployment of DPDK. >> If the power consumption incurred while running DPDK were a corner case, >> then there would be little to no research value in investigating such >> behavior. >> >> >> Your references don’t take into account the code that this community >> would actually deploy; open source implementations like DANOS, FD.io, or >> OVS. They don’t audit any commercial products that implement userspace >> stacks. None of your references say that DPDK is inherently inefficient. >> The closest they come is to say that tight polling is inefficient. But >> tight polling, even in the earliest days of DPDK, was never meant to be a >> design pattern that was actually deployed into production. I was there for >> those early conversations. >> >> Please don’t mislead the community into believing that DPDK == power bad >>> >> I have to object to this statement. It does seem to imply malice, or, at >> best, amateurish behaviour, whether you intended it or not. >> >> >> Object all you want. You are misleading people with your comments. And >> in the process you are denigrating a large swath of OSS projects and >> commercial products that use DPDK. Your survey questions are leading and >> provide a false dichotomy. And when you post the results here, they will >> be archived forever to continue to spread misinformation, unfortunately. >> >> Everything following is informational. Stop here if so inclined. >>> >> Please stop delving into the detail of DPDK's facilities without regard >> for your logical omission: >> that whether the facilities are available or not, DPDK's deployment >> profile (meaning: how it's being used in general), as indicated by the >> references I've provided, >> are leading to high power inefficiency on cores partitioned to the data >> plane. >> >> >> I’ve been writing network appliance code for over 20 years. I designed >> network architectures for years before that. I have 10s of thousands of >> DPDK based appliances in production at this moment across multiple >> different use cases. I work with companies that have 100s of thousands of >> units in production that leverage userspace runtimes. I do think I >> understand DPDK’s deployment profile better than you. That’s what I have >> been trying to tell you. People don’t write inefficient DPDK code to put >> into production. We’re not dumb. We’ve been thinking about power >> consumption from day one. DPDK was never supposed to be just a tight loop >> poll. You were always supposed to put in the very minimal extra work to >> modulate power consumption. >> >> The takeaway is that DPDK (and similar) doesn’t guarantee runaway power >>> bills. >>> >> Of course it doesn't. >> Even the second question of that bare-bones survey tried to communicate >> this much. >> >> If you have questions, I’d be happy to discuss off line >>> >> I would be happy to answer your objections in detail off line too. >> Just let me know. >> >> >> Unfortunately, you don’t seem to be receptive to the numerous people >> contradicting your assertions. So I’m out. I’ll let my comments stand >> here. >> >> Cheers, >> >> Etienne >> >> >> On Wed, Feb 24, 2021 at 12:12 AM Robert Bays <rob...@gdk.org> wrote: >> >>> Hi Etienne, >>> >>> Your statement that DPDK “keeps utilization at 100% regardless of packet >>> activity” is just not correct. You further pre-suppose "widespread DPDK's >>> core operating inefficiency” without any data to backup the operating >>> inefficacy assertion. Your statements, taken at face value, lead people to >>> believe that if a project uses DPDK it’s going to increase their power >>> costs. And that’s just not the case. Please don’t mislead the community >>> into believing that DPDK == power bad. >>> >>> Everything following is informational. Stop here if so inclined. >>> >>> DPDK does not dictate CPU utilization or power consumption, the >>> application leveraging DPDK does. It’s the application that decides how to >>> poll packets. If an application implements DPDK using only a tight polling >>> loop, then it will keep CPU cores that are running DPDK threads at 100%. >>> But only the most simple and/or bespoke (think trading) applications are >>> implemented this way. You don’t need tight polling all the time to get the >>> performance gains provided by DPDK or similar environments. The vast >>> majority of applications that this audience would actually install in their >>> networks do not do tight polling all the time and therefore don’t consume >>> 100% of the CPU all the time. An interesting, helpful research effort you >>> could lead would be to survey the ecosystem to catalog those applications >>> that do fall into the power hungry category and help them to change their >>> code. >>> >>> Intel DPDK application development guidelines don’t pre-suppose tight >>> polling all the time and offer at least two methods for optimizing power >>> against throughput. The older method is to use adaptive polling; >>> increasing the polling frequency as traffic load increases. This keeps cpu >>> utilization low when packet load is light and increases it as traffic >>> levels warrant. The second method is to use P-states and/or C-states to >>> put the processor into lower power modes when traffic loads are lighter. >>> We have found that adaptive polling works better across a larger pool of >>> hardware types, and therefore that is what DANOS uses, amongst other >>> things. >>> >>> Further, performance and power consumption are dictated by a >>> multivariate set of application decisions including: design patterns such >>> as single thread run to completion models vs. passing mbufs between >>> multiple threads, buffer sizes and cache management algorithms, combining >>> and/or separating tx/rx threads, binding threads to specific lcores, >>> reserved cores for DPDK threads, hyperthreading, kernel schedulers, >>> hypervisor schedulers, interface drivers, etc. All of these are >>> application specific, not DPDK generic. Well written applications that >>> leverage DPDK provide knobs for the user to tune these settings for their >>> specific environment and use case. None of this unique to DPDK. Solution >>> designs were cribbed from previous technologies. >>> >>> The takeaway is that DPDK (and similar) doesn’t guarantee runaway power >>> bills. Power consumption is dictated by the application. Look for well >>> behaved applications and everything will be alright. >>> >>> If you have questions, I’d be happy to discuss off line. >>> >>> Thanks, >>> Robert. >>> >>> >>> > On Feb 22, 2021, at 11:27 PM, Etienne-Victor Depasquale < >>> ed...@ieee.org> wrote: >>> > >>> > Sorry, last line should have been: >>> > "intended to get an impression of how widespread ***knowledge of*** >>> DPDK's core operating inefficiency is", >>> > not: >>> > "intended to get an impression of how widespread DPDK's core operating >>> inefficiency is" >>> > >>> > On Tue, Feb 23, 2021 at 8:22 AM Etienne-Victor Depasquale < >>> ed...@ieee.org> wrote: >>> > Beyond RX/TX CPU affinity, in DANOS you can further tune power >>> consumption by changing the adaptive polling rate. It doesn’t, per the >>> survey, "keep utilization at 100% regardless of packet activity.” >>> > Robert, you seem to be conflating DPDK >>> > with DANOS' power control algorithms that modulate DPDK's default >>> behaviour. >>> > >>> > Let me know what you think; otherwise, I'm pretty confident that DPDK >>> does: >>> > "keep utilization at 100% regardless of packet activity.” >>> > >>> > Keep in mind that this is a bare-bones survey intended for busy, >>> knowledgeable people (the ones you'd find on NANOG) - >>> > not a detailed breakdown of modes of operation of DPDK or DANOS. >>> > DPDK has been designed for fast I/O that's unencumbered by the >>> trappings of general-purpose OSes, >>> > and that's the impression that needs to be forefront. >>> > Power control, as well as any other dimensions of modulation, >>> > are detailed modes of operation that are well beyond the scope of a >>> bare-bones 2-question survey >>> > intended to get an impression of how widespread DPDK's core operating >>> inefficiency is. >>> > >>> > Cheers, >>> > >>> > Etienne >>> > >>> > On Mon, Feb 22, 2021 at 10:20 PM Robert Bays <rob...@gdk.org> wrote: >>> > Beyond RX/TX CPU affinity, in DANOS you can further tune power >>> consumption by changing the adaptive polling rate. It doesn’t, per the >>> survey, "keep utilization at 100% regardless of packet activity.” Adaptive >>> polling changes in DPDK optimize for tradeoffs between power consumption, >>> latency/jitter and drops during throughput ramp up periods. Ideally your >>> DPDK implementation has an algorithm that tries to automatically optimize >>> based on current traffic patterns. >>> > >>> > In DANOS refer to the “system default dataplane power-profile” config >>> command tree for adaptive polling settings. Interface RX/TX affinity is >>> configured on a per interface basis under the “interfaces dataplane” config >>> command tree. >>> > >>> > -robert >>> > >>> > >>> > > On Feb 22, 2021, at 11:46 AM, Jared Geiger <ja...@compuwizz.net> >>> wrote: >>> > > >>> > > DANOS lets you specify how many dataplane cores you use versus >>> control plane cores. So if you put a 16 core host in to handle 2GB of >>> traffic, you can adjust the dataplane worker cores as needed. Control plane >>> cores don't stay at 100% utilization. >>> > > >>> > > I use that technique plus DANOS runs on VMware (not oversubscribed) >>> which allows me to use the hardware for other VMs. NICS are attached to the >>> VM via PCI Passthrough which helps eliminate the overhead to the VMware >>> hypervisor itself. >>> > > >>> > > I have an 8 core VM with 4 cores set to dataplane and 4 to control >>> plane. The 4 control plane cores are typically idle only processing BGP >>> route updates, SNMP, logs, etc. >>> > > >>> > > ~Jared >>> > > >>> > > On Sun, Feb 21, 2021 at 11:30 PM Etienne-Victor Depasquale < >>> ed...@ieee.org> wrote: >>> > > Hello folks, >>> > > >>> > > I've just followed a thread regarding use of CGNAT and noted a >>> suggestion (regarding DANOS) that includes use of DPDK. >>> > > >>> > > As I'm interested in the breadth of adoption of DPDK, and as I'm a >>> researcher into energy and power efficiency, I'd love to hear your feedback >>> on your use of power consumption control by DPDK. >>> > > >>> > > I've drawn up a bare-bones, 2-question survey at this link: >>> > > >>> > > https://www.surveymonkey.com/r/J886DPY. >>> > > >>> > > Responses have been set to anonymous. >>> > > >>> > > Cheers, >>> > > >>> > > Etienne >>> > > >>> > > -- >>> > > Ing. Etienne-Victor Depasquale >>> > > Assistant Lecturer >>> > > Department of Communications & Computer Engineering >>> > > Faculty of Information & Communication Technology >>> > > University of Malta >>> > > Web. https://www.um.edu.mt/profile/etiennedepasquale >>> > >>> > >>> > >>> > -- >>> > Ing. Etienne-Victor Depasquale >>> > Assistant Lecturer >>> > Department of Communications & Computer Engineering >>> > Faculty of Information & Communication Technology >>> > University of Malta >>> > Web. https://www.um.edu.mt/profile/etiennedepasquale >>> > >>> > >>> > -- >>> > Ing. Etienne-Victor Depasquale >>> > Assistant Lecturer >>> > Department of Communications & Computer Engineering >>> > Faculty of Information & Communication Technology >>> > University of Malta >>> > Web. https://www.um.edu.mt/profile/etiennedepasquale >>> >>> >> >> -- >> Ing. Etienne-Victor Depasquale >> Assistant Lecturer >> Department of Communications & Computer Engineering >> Faculty of Information & Communication Technology >> University of Malta >> Web. https://www.um.edu.mt/profile/etiennedepasquale >> >> >> > > -- > Ing. Etienne-Victor Depasquale > Assistant Lecturer > Department of Communications & Computer Engineering > Faculty of Information & Communication Technology > University of Malta > Web. https://www.um.edu.mt/profile/etiennedepasquale > -- Ing. Etienne-Victor Depasquale Assistant Lecturer Department of Communications & Computer Engineering Faculty of Information & Communication Technology University of Malta Web. https://www.um.edu.mt/profile/etiennedepasquale
