Re: HTTPS LB and x-forwarded-for
Wido, backend servers are not Linux only, for example we have a ton of Windows customers, some WEB solutions / IIS etc... @all - If we try to please/solve everyone's proxying solution/requirement - this is impossible IMHO - I'm thinking more about some "do it as you like" solution, to let customer write his own haproxy config and upoad it (for example, or something better?). We can support newer version of haproxy (1.5+) which also implement "transarent proxy" (integrate with kernel so to speak) to allow TCP-level connections to backend (TCP mode, not HTTP mode) but to still "preserve" remote IP by faking it (fake soruce IP = transarent proxy). For the rest of configuration options, I would leave it to the customer how he/she wants to configure rest of haproxy configuration, inlcuding custom checks, etc. Haproxy configuration is never-ending story, and we probably should allow custom sripts/configuration instead of trying to provide GUI/API way to configure everything (which is impossible...) Just my 2 cents... On 9 November 2017 at 08:13, Wido den Hollander wrote: > > > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion >: > > > > > > Same challenge here too! > > > > Let's look at improving Load-balancing offering from cloudstack, I guest > we > > should do a feature spec draft soon.., from my perspective, doing SSL > > offload on the VR could be problematic if the VR spec if too small, and > the > > default spec of the VR being 1vcpu@256MB, considering it can be the > router > > of a VPC, doing VPN termination, adding HTTPS is a bit ish... What would > > be your thought about this ? > > > > I'd be curious to have a LB offering in ACS where it would deploy a > > redundant traefik[1] beside the VR for doing http and https > Load-balancing. > > I think it would also be useful if the API of that traefik instance would > > be available from within the VPC or LBnetwork so is API would be > accessible > > to other apps orchestration tools such as kubernetes or rancher. > > > > traefik or not, here is what I think is needed by cloudstack in the LB > > improvement: > > > > - support http, https (X-Forwarded-For) > > HAProxy also supports the PROXY protocol towards the backends. Apache > 2.4.22 supports this natively and Varnish for example can also talk PROXY. > > It adds a littlebit of metadata to the connection so that the backend > knows the original IP the connection came from for example: > https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt > > Wido > > > - basic persistence tuning (API already exist) > > - better backend monitoring, currently only a tcp connect validate if the > > webserver is up. > > - ssl offload > > - metric collection, more stats, maybe just export the tool status page > to > > the private network. > > - Container world support, right now if you have Rancher or kubernetes > > cluster, you need to deploy your own LB solution behing mostlikely a > static > > nat., If cloudstack would deploy a traefik instance, Kub or Rancher could > > reuse this instance and managed it to properly do LB between containers. > > > > > > What would be your prefered LB tool: > > haproxy, traefik or nginx? > > > > CloudStack already have to code to handle SSL certs per projects and > > accounts if not mistaking because that code was added to support > NetScaler > > as Load-balancer in the past. so one less thing to think about :-) > > > > > > [1] https://traefik.io/ > > > > > > PL, > > > > > > > > On Mon, Nov 6, 2017 at 7:10 AM, Nux! wrote: > > > > > Thanks Andrija, > > > > > > LB outside of the VR sounds like a good idea. An appliance based on, > say > > > cloud-init + ansible and so on could do the trick; alas it'd need to be > > > outside ACS. > > > I guess as users we could maybe come up with a spec for an > improvement, at > > > least we'd have something the devs could look at whenever it is > possible. > > > > > > Regards, > > > Lucian > > > > > > -- > > > Sent from the Delta quadrant using Borg technology! > > > > > > Nux! > > > www.nux.ro > > > > > > - Original Message - > > > > From: "Andrija Panic" > > > > To: "dev" > > > > Cc: "users" > > > > Sent: Thursday, 2 November, 2017 23:21:37 > > > > Subject: Re: HTTPS LB and x-forwarded-for > > > > > > > We used to make some special stuff for one of the clients, where all > LB > > > > configuration work is done from outside of the ACS, i.e. python > script to > > > > feed/configure VR - install latest haproxy 1.5.x for transparent > proxy, > > > > since client insisted on SSL termination done on backend web SSL > > > servers > > > > Not good idea, that is all I can say (custom configuration thing) - > but > > > the > > > > LB setup is actually good - transparent mode haproxy, works on TCP > level, > > > > so you can see "real client IP" on the backend servers (which must > use VR > > > > as the default gtw, as per default, so the whole setup works > properly). > > > > > > > > I'm still looking forward to see some spec
Please merge DOCs
https://github.com/apache/cloudstack-docs/pull/22 " Default value of "net.ipv4.igmp_max_memberships" (cat /proc/sys/net/ipv4/igmp_max_memberships) is "20", which means that host can be joined to max 20 multicast groups (attach max 20 multicast IPs on the host). Since all VXLAN (VTEP) interfaces provisioned on host are multicast-based (belong to certain multicast group, and thus has it's own multicast IP that is used as VTEP), this means that you can not provision more than 20 (working) VXLAN interfaces per host. On Linux kernel 3.x you actually can provision more than 20, but ARP request will silently fail and cause client's networking problems On Linux kernel 4.x you can NOT provision (start) more than 20 VXLAN interfaces and error message "No buffer space available" can be observed in Cloudstack Agent logs after provisioning required bridges and VXLAN interfaces. Increase needed parameter to sane value (i.e. 100 or 200) as required. If you need to operate more than 20 VMs from different client's network, this change above is required. " -- Andrija Panić
Re: HTTPS LB and x-forwarded-for
Wido, Excellent suggestion with the "transparent proxy", I was not aware of that. I think that would be a great idea and wouldn't require too many modifications, especially as Haproxy comes already with the VR. To Paul: - imho the LB solution ACS ships now is a bit handicaped since you do not know the remote host ip. You're flying blind unless you use google analytics (and these things have gotten more and more aggressively filtered by adblocks). Enhancing Haproxy as Wido suggested would go a long way, it wouldn't break existing functionality and would also keep SSL processing off the VR. -- Sent from the Delta quadrant using Borg technology! Nux! www.nux.ro - Original Message - > From: "Andrija Panic" > To: "users" > Cc: "Khosrow Moossavi" , "Will Stevens" > , "dev" > , "Pierre-Luc Dion" > Sent: Thursday, 9 November, 2017 13:10:58 > Subject: Re: HTTPS LB and x-forwarded-for > Wido, > > backend servers are not Linux only, for example we have a ton of Windows > customers, some WEB solutions / IIS etc... > > @all - If we try to please/solve everyone's proxying solution/requirement - > this is impossible IMHO - I'm thinking more about some "do it as you like" > solution, to let customer write his own haproxy config and upoad it (for > example, or something better?). > > We can support newer version of haproxy (1.5+) which also implement > "transarent proxy" (integrate with kernel so to speak) to allow TCP-level > connections to backend (TCP mode, not HTTP mode) but to still "preserve" > remote IP by faking it (fake soruce IP = transarent proxy). > > For the rest of configuration options, I would leave it to the customer > how he/she wants to configure rest of haproxy configuration, inlcuding > custom checks, etc. Haproxy configuration is never-ending story, and we > probably should allow custom sripts/configuration instead of trying to > provide GUI/API way to configure everything (which is impossible...) > > Just my 2 cents... > > On 9 November 2017 at 08:13, Wido den Hollander wrote: > >> >> > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion > >: >> > >> > >> > Same challenge here too! >> > >> > Let's look at improving Load-balancing offering from cloudstack, I guest >> we >> > should do a feature spec draft soon.., from my perspective, doing SSL >> > offload on the VR could be problematic if the VR spec if too small, and >> the >> > default spec of the VR being 1vcpu@256MB, considering it can be the >> router >> > of a VPC, doing VPN termination, adding HTTPS is a bit ish... What would >> > be your thought about this ? >> > >> > I'd be curious to have a LB offering in ACS where it would deploy a >> > redundant traefik[1] beside the VR for doing http and https >> Load-balancing. >> > I think it would also be useful if the API of that traefik instance would >> > be available from within the VPC or LBnetwork so is API would be >> accessible >> > to other apps orchestration tools such as kubernetes or rancher. >> > >> > traefik or not, here is what I think is needed by cloudstack in the LB >> > improvement: >> > >> > - support http, https (X-Forwarded-For) >> >> HAProxy also supports the PROXY protocol towards the backends. Apache >> 2.4.22 supports this natively and Varnish for example can also talk PROXY. >> >> It adds a littlebit of metadata to the connection so that the backend >> knows the original IP the connection came from for example: >> https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt >> >> Wido >> >> > - basic persistence tuning (API already exist) >> > - better backend monitoring, currently only a tcp connect validate if the >> > webserver is up. >> > - ssl offload >> > - metric collection, more stats, maybe just export the tool status page >> to >> > the private network. >> > - Container world support, right now if you have Rancher or kubernetes >> > cluster, you need to deploy your own LB solution behing mostlikely a >> static >> > nat., If cloudstack would deploy a traefik instance, Kub or Rancher could >> > reuse this instance and managed it to properly do LB between containers. >> > >> > >> > What would be your prefered LB tool: >> > haproxy, traefik or nginx? >> > >> > CloudStack already have to code to handle SSL certs per projects and >> > accounts if not mistaking because that code was added to support >> NetScaler >> > as Load-balancer in the past. so one less thing to think about :-) >> > >> > >> > [1] https://traefik.io/ >> > >> > >> > PL, >> > >> > >> > >> > On Mon, Nov 6, 2017 at 7:10 AM, Nux! wrote: >> > >> > > Thanks Andrija, >> > > >> > > LB outside of the VR sounds like a good idea. An appliance based on, >> say >> > > cloud-init + ansible and so on could do the trick; alas it'd need to be >> > > outside ACS. >> > > I guess as users we could maybe come up with a spec for an >> improvement, at >> > > least we'd have something the devs could look at whenever it is >> possible. >> > > >> > > Regards, >> > > Lucian >> > > >> > > -- >>
Re: Strange size of template from snapshot on XenServer
Anshul and Lotic, after debugging and inspecting code for quite a while I understood the complete picture and created a solution for it. The PR with the solution is found athttps://github.com/apache/cloudstack/pull/2315 . This was a clear example of what I constantly tell people here, code needs to be clear, concise, well tested and documented. Here follows some explanation regarding the removal of some code introduced with commit “2c4ea503f92bcf9c611f409d5cdecb”. With commit “8caf52c” Daan and I removed a small piece of code introduced by “2c4ea503f92bcf9c611f409d5cdecb” because it did not seem to make much sense. We did not remove everything; we only removed the part that was looking for a random host in the zone to execute the command. As we (Anshul) exchanged some messages in some other PR, the code introduced in “2c4ea503f92bcf9c611f409d5cdecb” would only work for zones that do not have other hypervisor types if they are deployed with XenServer clusters.It was being created a limitation in ACS that should not exist. I stress again that this only happened for the lack of documentation and clear coding. For instance, when I read the commit “2c4ea503f92bcf9c611f409d5cdecb”, it says it introduced an “optimization”, then I assumed that the process executed with our code base before commit “2c4ea503f92bcf9c611f409d5cdecb” is working, but not as fast or with as much quality as the code with “2c4ea503f92bcf9c611f409d5cdecb”. However, that is not the case; the code “2c4ea503f92bcf9c611f409d5cdecb” is not optimizing anything; it is, in fact, fixing/creating a workflow to create templates from snapshots in XenServer deployments. The first PR(#1176) intended to solve #CLOUDSTACK-9025 was only tackling the problem for CloudStack deployments that use single hypervisor types (restricted to XenServers host in the same zone; this means, it was not expecting to have multiple hypervisors types in the same zone). Additionally, the lack of information regarding that solution (documentation, test cases and description in PRs and Jira ticket) led the code to be removed in #1124 after a long discussion and analysis in #1056. That piece of code seemed logicless. It would receive a hostId and then change that hostId for other hostId of the zone without doing any check; it was not even checking the hypervisor and storage in which the host was connected to. The problem reported in #CLOUDSTACK-9025 is caused by partial snapshots that are taken in XenServer. This means, we do not take a complete snapshot, but a partial one that contains only the modified data. This requires rebuilding the VHD hierarchy when creating a template out of the snapshot. The point is that the first hostId received is not a hostId, but a system VM ID(SSVM). That is why the code in #1176 fixed the problem for some deployment scenarios, but would cause problems for scenarios where we have multiple hypervisors in the same zone. We need to execute the creation of the VHD that represents the template in the hypervisor, so the VHD chain can be built using the parent links. The PR #2315 changes the behavior of com.cloud.hypervisor.XenServerGuru.getCommandHostDelegation(long, Command). >From now on we replace the hostId that is intended to execute the “copy command” that will create the VHD of the template according to some conditions that were already in place. The idea is that starting with XenServer 6.2.0 hotFix ESP1004 we need to execute the command in the hypervisor host and not from the SSVM. Moreover, the method was improved making it more readable; it was also created test cases assuring that from XenServer 6.2.0 hotFix ESP1004 and upward versions we change the hostId that will be used to execute the “copy command”. Furthermore, we are not selecting a random host from a zone anymore. A new method was introduced in the HostDao object called “findHostConnectedToSnapshotStoragePoolToExecuteCommand”, using this method we look for a host that is in the cluster that is using the storage pool where the volume from which the Snapshot is taken of resides. By doing this, we guarantee that the host that is connected to the primary storage where all of the snapshots parent VHDs are stored is used to create the template. Also, I opened the PR only against master. If someone needs the fix in any other version we can cherry pick to 4.9.2.0. On Wed, Nov 8, 2017 at 9:41 AM, Anshul Gangwar < anshul.gang...@accelerite.com> wrote: > Rafael, In ACS 4.5.2 you are facing issue due to XenServer version (was > only working on xenserver 6.2 with some patches) which got fixed in 4.6 > with commit 2c4ea503f92bcf9c611f409d5cdecb42b0115b69. Now with code clean > up in commit “8a3fd10” it’s broken. During storage refactoring changes in > commit “8caf52c” optimization was done to send CopyCommand to Hypervisor > resource instead of SSVM. Now with commit “8a3fd10” it’s again going to > SSVM so it’s working fine for first snapshot but broken for linked snapshot. > > Regards, > An
Re: HTTPS LB and x-forwarded-for
> Op 9 november 2017 om 19:59 schreef Nux! : > > > Wido, > > Excellent suggestion with the "transparent proxy", I was not aware of that. > I think that would be a great idea and wouldn't require too many > modifications, especially as Haproxy comes already with the VR. > It's indeed just a matter of a HAProxy config setting. We could make it configurable per backend in HAProxy. Regular HTTP, TCP or PROXY for example. That way your problem would be solved. Wido > To Paul: > - imho the LB solution ACS ships now is a bit handicaped since you do not > know the remote host ip. You're flying blind unless you use google analytics > (and these things have gotten more and more aggressively filtered by > adblocks). > Enhancing Haproxy as Wido suggested would go a long way, it wouldn't break > existing functionality and would also keep SSL processing off the VR. > > -- > Sent from the Delta quadrant using Borg technology! > > Nux! > www.nux.ro > > - Original Message - > > From: "Andrija Panic" > > To: "users" > > Cc: "Khosrow Moossavi" , "Will Stevens" > > , "dev" > > , "Pierre-Luc Dion" > > Sent: Thursday, 9 November, 2017 13:10:58 > > Subject: Re: HTTPS LB and x-forwarded-for > > > Wido, > > > > backend servers are not Linux only, for example we have a ton of Windows > > customers, some WEB solutions / IIS etc... > > > > @all - If we try to please/solve everyone's proxying solution/requirement - > > this is impossible IMHO - I'm thinking more about some "do it as you like" > > solution, to let customer write his own haproxy config and upoad it (for > > example, or something better?). > > > > We can support newer version of haproxy (1.5+) which also implement > > "transarent proxy" (integrate with kernel so to speak) to allow TCP-level > > connections to backend (TCP mode, not HTTP mode) but to still "preserve" > > remote IP by faking it (fake soruce IP = transarent proxy). > > > > For the rest of configuration options, I would leave it to the customer > > how he/she wants to configure rest of haproxy configuration, inlcuding > > custom checks, etc. Haproxy configuration is never-ending story, and we > > probably should allow custom sripts/configuration instead of trying to > > provide GUI/API way to configure everything (which is impossible...) > > > > Just my 2 cents... > > > > On 9 November 2017 at 08:13, Wido den Hollander wrote: > > > >> > >> > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion >> >: > >> > > >> > > >> > Same challenge here too! > >> > > >> > Let's look at improving Load-balancing offering from cloudstack, I guest > >> we > >> > should do a feature spec draft soon.., from my perspective, doing SSL > >> > offload on the VR could be problematic if the VR spec if too small, and > >> the > >> > default spec of the VR being 1vcpu@256MB, considering it can be the > >> router > >> > of a VPC, doing VPN termination, adding HTTPS is a bit ish... What would > >> > be your thought about this ? > >> > > >> > I'd be curious to have a LB offering in ACS where it would deploy a > >> > redundant traefik[1] beside the VR for doing http and https > >> Load-balancing. > >> > I think it would also be useful if the API of that traefik instance would > >> > be available from within the VPC or LBnetwork so is API would be > >> accessible > >> > to other apps orchestration tools such as kubernetes or rancher. > >> > > >> > traefik or not, here is what I think is needed by cloudstack in the LB > >> > improvement: > >> > > >> > - support http, https (X-Forwarded-For) > >> > >> HAProxy also supports the PROXY protocol towards the backends. Apache > >> 2.4.22 supports this natively and Varnish for example can also talk PROXY. > >> > >> It adds a littlebit of metadata to the connection so that the backend > >> knows the original IP the connection came from for example: > >> https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt > >> > >> Wido > >> > >> > - basic persistence tuning (API already exist) > >> > - better backend monitoring, currently only a tcp connect validate if the > >> > webserver is up. > >> > - ssl offload > >> > - metric collection, more stats, maybe just export the tool status page > >> to > >> > the private network. > >> > - Container world support, right now if you have Rancher or kubernetes > >> > cluster, you need to deploy your own LB solution behing mostlikely a > >> static > >> > nat., If cloudstack would deploy a traefik instance, Kub or Rancher could > >> > reuse this instance and managed it to properly do LB between containers. > >> > > >> > > >> > What would be your prefered LB tool: > >> > haproxy, traefik or nginx? > >> > > >> > CloudStack already have to code to handle SSL certs per projects and > >> > accounts if not mistaking because that code was added to support > >> NetScaler > >> > as Load-balancer in the past. so one less thing to think about :-) > >> > > >> > > >> > [1] https://traefik.io/ > >> > > >> > > >> > PL, > >> > > >> > > >
experienced Apache CloudStack Developer
Hi Do you know some skilled and experienced Apache CloudStack Developer, how likes to work for a successful Swiss Cloud Service Provider company? If so, please send the application to cl...@1st.ch. Of course the work would be contributed to the community (no forc). Thx Regards Heinz
