>From a certain point of view, OpenWRT is most likely not suitable for our needs, this project is highly oriented for router appliance platforms and kernel is most likely highly optimised for a small memory footprint and might not have performance and virtualization in mind.
It's a shame that there is not some sort of API driven open source VyOS :-S Anything useful that would come out of the CloudNative projects? On Fri, Aug 13, 2021 at 5:48 AM Rohit Yadav <rohit.ya...@shapeblue.com> wrote: > Hi Pierre-Luc, > > Thanks for replying. > > We've another thread on-going to discuss IPv6 support where we've indeed > discussed idea of introducing dynamic routing capabilities in the VR in a > future phase with FRR in the VR. > > For the VR agent idea, I'm indeed thinking of API driven way (the agent > exposes an API for CloudStack and a custom CLI perhaps) for it to manage > iptables, software packages/services such as dnsmasq etc. Right now it's > possible to configure some of the services in network offering (for ex. if > you don't want LB etc). > > I've indeed looked at some alternatives: > > * Alpine: I experimented a new systemvmtemplate based on Alpine ( > https://github.com/shapeblue/alpine-systemvm) but we saw many cons with > Alpine and the gains weren't significant so we decided to drop the idea of > moving from Debian to Alpine. > * VyOS: I've concerns about project and community (I could be wrong) > to have our VR depend on it as the "default" provider; though I see an old > PR > * OpenWRT: Among available router distributions, I really liked it - > it has (a) a good health and active community and commit activity ( > https://github.com/openwrt/openwrt/graphs/contributors), (b) multi-years > (10+yrs) of releases (https://openwrt.org/about/history), (c) a > configuration system (UCI) and UI (LUCI) which can be used to manage > services, interfaces, firewall, routes etc with a Lua based API, (d) opkg > packaging system, most software packages or equivalent are available that > our Debian based systemvmtemplate needs/uses (java jre workaround from > Alpine though), (e) good documentation on extending features/writing > plugins etc, (f) they tend to do regular releases and use Linux/LTS kernel. > * pfSense/opnSense: the major cons are FreeBSD-based kernel and skills > that CloudStack community may not have in terms of debugging, investigation > etc like they have for Linux > > Among the above, OpenWRT is probably worth exploring in my opinion; other > some firewall/iptables and iproute2 (interfaces and routes) API/library > that can be used by the VR agent on Debian to program the firewall (acls, > firewall, pf), IP/interface management and routing features in the VR > (currently it's all done in a manual sense which is untested and often > source of bugs and issues). > > Lastly, we already have this PR which aims to do automatic > systemvmtemplate registeration/installation/seeding and handle upgrades > (targetted for 4.16): https://github.com/apache/cloudstack/pull/4329 The > PR also unifies the systemvm template use to be extended for CKS use-case > (i.e. no separate template installation required for CKS). > > > Regards. > > ________________________________ > From: Pierre-Luc Dion <pdion...@apache.org> > Sent: Thursday, August 12, 2021 16:45 > To: dev <dev@cloudstack.apache.org> > Subject: Re: [DISCUSS] NextGen VR? > > Hi Rohit, > > Like it! Our VR system is due for some rethinking! > I don't have much points to add to issues you highlight, it seems pretty > complete. > > Here are some more features or ideas that would be interesting to see in a > new VR system: > - Use or support for routing protocol, such as BGP or OSBP so we could > provide a more dynamic PrivateGateway concept. using FRR[1]? > - have an API driven way to configure IPtables and other network services. > - could we decouple network services such as LB, VPNserver, gateways from > the VR ? > > Debian has been a pain for building VR because the iso defined in our > config need constant update, but on the other hand it's been proved to be a > reliable OS, we saw some of our VR with uptime over 1000 days. > > As an alternative OS, I'd be curious to look at VyOS[2] or Alpine Linux. > > From a certain perspective, us providing the systemvm template make sure > that systemVM will deploy. work reliably and make it a single template to > test. Compared to a system that would just provide RPM/DEB packages and > mechanism to push configs, this could require to test all kinds of template > scenarios, since users could use any version of distro to deploy their > systemVM/VRs. > > "Users forget to register the right systemvmtemplate for a new ACS version > and upgrades fail": Maybe we could automatically register the new template > during the upgrade process? This feature used to exist in the Citrix > CloudPlatform. > > > [1] https://frrouting.org/ > [2] https://github.com/vyos/vyos-1x > > On Wed, Aug 11, 2021 at 12:37 PM Rohit Yadav <rohit.ya...@shapeblue.com> > wrote: > > > All, > > > > We've over the years create a VR that largely is stable but we've > > discussed the pain of maintaining, extending, and upgrading VRs both in > > lists and in user-groups and CCC conferences. > > > > On a high-level the pain points are: > > > > * It's difficult to debug, investigate VR for operators and support > > team > > * Maintaining the VR code, fixing bugs, implementing features is a > > pain for developers; further the xml&json databags based programming > model > > is confusing > > * Any fix or changes requires a new systemvmtemplate or VR codebase > > whose patching requires restarting the VR or destroying an old VR > > * No uniform VR programming API (current approach is SSH+databags and > > execute a script), this makes testing VR difficult and QA in isolation is > > not possible > > * Users forget to register the right systemvmtemplate for a new ACS > > version and upgrades fail > > * Others (please share yours)? > > > > Among these pain points my colleagues have proposed a PR targeting 4.16 > > [1] that aims to unify systemvmtemplate as a building block that is > bundled > > as part of ACS rpm/deb/* packages which CloudStack will automatically > > seed/register/use with which upgrades will be as simple as a yum update > or > > apt-get upgrade. Further, my colleagues and I are exploring a live patch > > API which in near future that can patch a running systemvm/VR using > > systemvm.iso (or deprecate systemvm.iso and use ssh/scp to patch?) > without > > requiring to reboot/recreate it. Hopefully, this addresses some of those > > pain points. We request the community for your feedback and > > review/participation in the PR. > > > > Open questions, topics to discuss and gather feedback: > > > > * VR programming: Should we explore a new light-weight VR agent that > > provides an API (restful/grpc, or CLI?), some mechanism of live patching > VR > > code, packages, and kernel? > > * Refactoring isolated network and VPC codebases into a unified > > codebase and feature sets (assumption that isolated network are largely a > > VPC with single tier), does it benefit the community, users, and > developers? > > * Underlying OS: > > * should we consider something other than Debian, any suggestions? > > * or explore stable/widely used and maintained opensource router > > distributions such as OpenWRT [2] which ships with a UI and > > CLI/configuration system UCI [3]? The cons of the approach are a new > > dependency and some likely missing packages. > > * In current VR codebase, most of the effort is spent in > > implementing/maintaining router packages/configure codebase which we can > > get rid of by depending on a stable Linux router distro which ships with > > some API/config system. Not choosing an existing router distribution > means > > we continue to DIY router programming+config management codebase. > > * Any other ideas? > > > > Thoughts, feedback? > > > > [1] https://github.com/apache/cloudstack/pull/4329 > > [2] https://github.com/openwrt/openwrt/graphs/contributors > > [3] https://openwrt.org/docs/guide-user/base-system/uci > > > > > > Regards. > > > > > > > > > > > >
