Hi Rohit, So look like OpenWRT could be a good alternative then, but how would you handle other services such as load-balancing, vpn,...? would that be from another Service VMs, kind of doing service mesh ?
Cheers, On Mon, Aug 16, 2021 at 5:37 AM Rohit Yadav <rohit.ya...@shapeblue.com> wrote: > Hi PL, > > I tested OpenWRT on KVM, and I don't see any performance loss in terms of > bandwidth/IO and memory, in fact it seems to be faster than the Debian > based systemvmtemplate due to: > > * Smaller kernel size (the Debian kernel includes all sorts of device > drivers for sound and what not which is not necessary for VR/systemvm) > * Faster ssh (the dropbear ssh kernel is on-par with openssh-server > but has smaller footprint and missing features such as sftp that we don't > need in VR/systemvm; the ssh and programming seemed really fast compared to > Debian's openssh-server) > * Really low memory consumption (Debian10+ eats up some RAM due to > non-optimised services, esp systemd compared to OpenWRT) > * Lua based really-fast APIs and bus (openwrt has > https://openwrt.org/docs/techref/ubus, I'm looking into how we can build > something similar for our Debian based systemvmtemplate if migrating to > OpenWRT is going to be a pain or unacceptable by the community) > > Based on your comment I found that VyOS rolling release has API support ( > https://blog.vyos.io/vyos-rolling-release-has-got-an-http-api) however > that is probably not supported for their lts/legacy release, we'll also > face issue with installing/configuring custom packages (java etc) that we > need for systemvm/VR. > > > Regards. > > ________________________________ > From: Pierre-Luc Dion <pdion...@apache.org> > Sent: Friday, August 13, 2021 21:51 > To: dev <dev@cloudstack.apache.org> > Subject: Re: [DISCUSS] NextGen VR? > > From a certain point of view, OpenWRT is most likely not suitable for our > needs, this project is highly oriented for router appliance platforms and > kernel is most likely highly optimised for a small memory footprint and > might not have performance and virtualization in mind. > > It's a shame that there is not some sort of API driven open source VyOS :-S > > Anything useful that would come out of the CloudNative projects? > > > On Fri, Aug 13, 2021 at 5:48 AM Rohit Yadav <rohit.ya...@shapeblue.com> > wrote: > > > Hi Pierre-Luc, > > > > Thanks for replying. > > > > We've another thread on-going to discuss IPv6 support where we've indeed > > discussed idea of introducing dynamic routing capabilities in the VR in a > > future phase with FRR in the VR. > > > > For the VR agent idea, I'm indeed thinking of API driven way (the agent > > exposes an API for CloudStack and a custom CLI perhaps) for it to manage > > iptables, software packages/services such as dnsmasq etc. Right now it's > > possible to configure some of the services in network offering (for ex. > if > > you don't want LB etc). > > > > I've indeed looked at some alternatives: > > > > * Alpine: I experimented a new systemvmtemplate based on Alpine ( > > https://github.com/shapeblue/alpine-systemvm) but we saw many cons with > > Alpine and the gains weren't significant so we decided to drop the idea > of > > moving from Debian to Alpine. > > * VyOS: I've concerns about project and community (I could be wrong) > > to have our VR depend on it as the "default" provider; though I see an > old > > PR > > * OpenWRT: Among available router distributions, I really liked it - > > it has (a) a good health and active community and commit activity ( > > https://github.com/openwrt/openwrt/graphs/contributors), (b) multi-years > > (10+yrs) of releases (https://openwrt.org/about/history), (c) a > > configuration system (UCI) and UI (LUCI) which can be used to manage > > services, interfaces, firewall, routes etc with a Lua based API, (d) opkg > > packaging system, most software packages or equivalent are available that > > our Debian based systemvmtemplate needs/uses (java jre workaround from > > Alpine though), (e) good documentation on extending features/writing > > plugins etc, (f) they tend to do regular releases and use Linux/LTS > kernel. > > * pfSense/opnSense: the major cons are FreeBSD-based kernel and > skills > > that CloudStack community may not have in terms of debugging, > investigation > > etc like they have for Linux > > > > Among the above, OpenWRT is probably worth exploring in my opinion; other > > some firewall/iptables and iproute2 (interfaces and routes) API/library > > that can be used by the VR agent on Debian to program the firewall (acls, > > firewall, pf), IP/interface management and routing features in the VR > > (currently it's all done in a manual sense which is untested and often > > source of bugs and issues). > > > > Lastly, we already have this PR which aims to do automatic > > systemvmtemplate registeration/installation/seeding and handle upgrades > > (targetted for 4.16): https://github.com/apache/cloudstack/pull/4329 The > > PR also unifies the systemvm template use to be extended for CKS use-case > > (i.e. no separate template installation required for CKS). > > > > > > Regards. > > > > ________________________________ > > From: Pierre-Luc Dion <pdion...@apache.org> > > Sent: Thursday, August 12, 2021 16:45 > > To: dev <dev@cloudstack.apache.org> > > Subject: Re: [DISCUSS] NextGen VR? > > > > Hi Rohit, > > > > Like it! Our VR system is due for some rethinking! > > I don't have much points to add to issues you highlight, it seems pretty > > complete. > > > > Here are some more features or ideas that would be interesting to see in > a > > new VR system: > > - Use or support for routing protocol, such as BGP or OSBP so we could > > provide a more dynamic PrivateGateway concept. using FRR[1]? > > - have an API driven way to configure IPtables and other network > services. > > - could we decouple network services such as LB, VPNserver, gateways > from > > the VR ? > > > > Debian has been a pain for building VR because the iso defined in our > > config need constant update, but on the other hand it's been proved to > be a > > reliable OS, we saw some of our VR with uptime over 1000 days. > > > > As an alternative OS, I'd be curious to look at VyOS[2] or Alpine Linux. > > > > From a certain perspective, us providing the systemvm template make sure > > that systemVM will deploy. work reliably and make it a single template to > > test. Compared to a system that would just provide RPM/DEB packages and > > mechanism to push configs, this could require to test all kinds of > template > > scenarios, since users could use any version of distro to deploy their > > systemVM/VRs. > > > > "Users forget to register the right systemvmtemplate for a new ACS > version > > and upgrades fail": Maybe we could automatically register the new > template > > during the upgrade process? This feature used to exist in the Citrix > > CloudPlatform. > > > > > > [1] https://frrouting.org/ > > [2] https://github.com/vyos/vyos-1x > > > > On Wed, Aug 11, 2021 at 12:37 PM Rohit Yadav <rohit.ya...@shapeblue.com> > > wrote: > > > > > All, > > > > > > We've over the years create a VR that largely is stable but we've > > > discussed the pain of maintaining, extending, and upgrading VRs both in > > > lists and in user-groups and CCC conferences. > > > > > > On a high-level the pain points are: > > > > > > * It's difficult to debug, investigate VR for operators and support > > > team > > > * Maintaining the VR code, fixing bugs, implementing features is a > > > pain for developers; further the xml&json databags based programming > > model > > > is confusing > > > * Any fix or changes requires a new systemvmtemplate or VR codebase > > > whose patching requires restarting the VR or destroying an old VR > > > * No uniform VR programming API (current approach is SSH+databags > and > > > execute a script), this makes testing VR difficult and QA in isolation > is > > > not possible > > > * Users forget to register the right systemvmtemplate for a new ACS > > > version and upgrades fail > > > * Others (please share yours)? > > > > > > Among these pain points my colleagues have proposed a PR targeting 4.16 > > > [1] that aims to unify systemvmtemplate as a building block that is > > bundled > > > as part of ACS rpm/deb/* packages which CloudStack will automatically > > > seed/register/use with which upgrades will be as simple as a yum update > > or > > > apt-get upgrade. Further, my colleagues and I are exploring a live > patch > > > API which in near future that can patch a running systemvm/VR using > > > systemvm.iso (or deprecate systemvm.iso and use ssh/scp to patch?) > > without > > > requiring to reboot/recreate it. Hopefully, this addresses some of > those > > > pain points. We request the community for your feedback and > > > review/participation in the PR. > > > > > > Open questions, topics to discuss and gather feedback: > > > > > > * VR programming: Should we explore a new light-weight VR agent > that > > > provides an API (restful/grpc, or CLI?), some mechanism of live > patching > > VR > > > code, packages, and kernel? > > > * Refactoring isolated network and VPC codebases into a unified > > > codebase and feature sets (assumption that isolated network are > largely a > > > VPC with single tier), does it benefit the community, users, and > > developers? > > > * Underlying OS: > > > * should we consider something other than Debian, any > suggestions? > > > * or explore stable/widely used and maintained opensource router > > > distributions such as OpenWRT [2] which ships with a UI and > > > CLI/configuration system UCI [3]? The cons of the approach are a new > > > dependency and some likely missing packages. > > > * In current VR codebase, most of the effort is spent in > > > implementing/maintaining router packages/configure codebase which we > can > > > get rid of by depending on a stable Linux router distro which ships > with > > > some API/config system. Not choosing an existing router distribution > > means > > > we continue to DIY router programming+config management codebase. > > > * Any other ideas? > > > > > > Thoughts, feedback? > > > > > > [1] https://github.com/apache/cloudstack/pull/4329 > > > [2] https://github.com/openwrt/openwrt/graphs/contributors > > > [3] https://openwrt.org/docs/guide-user/base-system/uci > > > > > > > > > Regards. > > > > > > > > > > > > > > > > > > > > > > > >
