Re: Strange size of template from snapshot on XenServer

[Anshul Gangwar]

Rafael, you are still misunderstanding the fix 
“2c4ea503f92bcf9c611f409d5cdecb”. That has only fixed the issue due to check 
for specific version i.e. before “2c4ea503f92bcf9c611f409d5cdecb” things were 
fine for xenserver 6.2 but not working for 6.5. Commit doesn’t mention anything 
about optimization. Other than version change it reverts the previous commit 
which is introducing hypervisor specific check in core code.

Regarding documentation, “git history”is the best documentation. Code 
documentation may not make sense after some changes as it is often ignored. 

See the PR for more comments.

Regards,
Anshul 

On 10/11/17, 3:14 AM, "Rafael Weingärtner"  wrote:

Anshul and Lotic, after debugging and inspecting code for quite a while I
understood the complete picture and created a solution for it. The PR with
the solution is found athttps://github.com/apache/cloudstack/pull/2315 .
This was a clear example of what I constantly tell people here, code needs
to be clear, concise, well tested and documented. Here follows some
explanation regarding the removal of some code introduced with commit
“2c4ea503f92bcf9c611f409d5cdecb”.

With commit “8caf52c” Daan and I removed a small piece of code introduced
by “2c4ea503f92bcf9c611f409d5cdecb” because it did not seem to make much
sense. We did not remove everything; we only removed the part that was
looking for a random host in the zone to execute the command. As we
(Anshul) exchanged some messages in some other PR, the code introduced in
“2c4ea503f92bcf9c611f409d5cdecb” would only work for zones that do not have
other hypervisor types if they are deployed with XenServer clusters.It was
being created a limitation in ACS that should not exist. I stress again
that this only happened for the lack of documentation and clear coding.
For instance, when I read the commit “2c4ea503f92bcf9c611f409d5cdecb”, it
says it introduced an “optimization”, then I assumed that the process
executed with our code base before commit “2c4ea503f92bcf9c611f409d5cdecb”
is working, but not as fast or with as much quality as the code with
“2c4ea503f92bcf9c611f409d5cdecb”. However, that is not the case; the code
“2c4ea503f92bcf9c611f409d5cdecb” is not optimizing anything; it is, in
fact, fixing/creating a workflow to create templates from snapshots in
XenServer deployments.

The first PR(#1176) intended to solve #CLOUDSTACK-9025 was only tackling
the problem for CloudStack deployments that use single hypervisor types
(restricted to XenServers host in the same zone; this means, it was not
expecting to have multiple hypervisors types in the same zone).
Additionally, the lack of information regarding that solution
(documentation, test cases and description in PRs and Jira ticket) led the
code to be removed in #1124 after a long discussion and analysis in #1056.
That piece of code seemed logicless.  It would receive a hostId and then
change that hostId for other hostId of the zone without doing any check; it
was not even checking the hypervisor and storage in which the host was
connected to.

The problem reported in #CLOUDSTACK-9025 is caused by partial snapshots
that are taken in XenServer. This means, we do not take a complete
snapshot, but a partial one that contains only the modified data. This
requires rebuilding the VHD hierarchy when creating a template out of the
snapshot. The point is that the first hostId received is not a hostId, but
a system VM ID(SSVM). That is why the code in #1176 fixed the problem for
some deployment scenarios, but would cause problems for scenarios where we
have multiple hypervisors in the same zone. We need to execute the creation
of the VHD that represents the template in the hypervisor, so the VHD chain
can be built using the parent links.

The PR #2315 changes the behavior of
com.cloud.hypervisor.XenServerGuru.getCommandHostDelegation(long, Command).
From now on we replace the hostId that is intended to execute the “copy
command” that will create the VHD of the template according to some
conditions that were already in place. The idea is that starting with
XenServer 6.2.0 hotFix ESP1004 we need to execute the command in the
hypervisor host and not from the SSVM. Moreover, the method was improved
making it more readable; it was also created test cases assuring that from
XenServer 6.2.0 hotFix ESP1004 and upward versions we change the hostId
that will be used to execute the “copy command”.

Furthermore, we are not selecting a random host from a zone anymore. A new
method was introduced in the HostDao object called
“findHostConnectedToSnapshotStoragePoolToExecuteCommand”, using this method
we look for a host that is in the cluster that is using the storage pool
where the volume from which the Snapshot is tak

Re: Strange size of template from snapshot on XenServer

[Rafael Weingärtner]

Anshul,

I will not touch the merit of documentation in code, for now, this is
another discussion.  And regarding the “optimization”, you brought it up,
whenever I saw you discussing this issue, you would talk about some sort of
“optimization”… see
https://github.com/apache/cloudstack/pull/1124#issuecomment-326550060. At
least for me, that was quite confusing, and I was not seeing what you were
talking about.

If you diff the files, what broke your “solution” was the removal of the
following lines:

> EndPoint ep = endPointSelector.selectHypervisorHost(new ZoneScope(host.
> getDataCenterId()));
> host = hostDao.findById(ep.getId());
>

The conditional checks were maintained. We only removed something that we
seemed logicless.

With PR https://github.com/apache/cloudstack/pull/2315, I did not add
anything extra. I only re-worked the method to make it clear and more
readable (removed the if-inception). The only point I changed is that,
instead of using “selectOneHypervisorHostByZone”, I create a specific
method for that, which selects a host according to the volume of the
snapshot.

On Fri, Nov 10, 2017 at 8:21 AM, Anshul Gangwar <
anshul.gang...@accelerite.com> wrote:

> Rafael, you are still misunderstanding the fix “
> 2c4ea503f92bcf9c611f409d5cdecb”. That has only fixed the issue due to
> check for specific version i.e. before “2c4ea503f92bcf9c611f409d5cdecb”
> things were fine for xenserver 6.2 but not working for 6.5. Commit doesn’t
> mention anything about optimization. Other than version change it reverts
> the previous commit which is introducing hypervisor specific check in core
> code.
>
> Regarding documentation, “git history”is the best documentation. Code
> documentation may not make sense after some changes as it is often ignored.
>
> See the PR for more comments.
>
> Regards,
> Anshul
>
> On 10/11/17, 3:14 AM, "Rafael Weingärtner" 
> wrote:
>
> Anshul and Lotic, after debugging and inspecting code for quite a
> while I
> understood the complete picture and created a solution for it. The PR
> with
> the solution is found athttps://github.com/apache/cloudstack/pull/2315
> .
> This was a clear example of what I constantly tell people here, code
> needs
> to be clear, concise, well tested and documented. Here follows some
> explanation regarding the removal of some code introduced with commit
> “2c4ea503f92bcf9c611f409d5cdecb”.
>
> With commit “8caf52c” Daan and I removed a small piece of code
> introduced
> by “2c4ea503f92bcf9c611f409d5cdecb” because it did not seem to make
> much
> sense. We did not remove everything; we only removed the part that was
> looking for a random host in the zone to execute the command. As we
> (Anshul) exchanged some messages in some other PR, the code introduced
> in
> “2c4ea503f92bcf9c611f409d5cdecb” would only work for zones that do
> not have
> other hypervisor types if they are deployed with XenServer clusters.It
> was
> being created a limitation in ACS that should not exist. I stress again
> that this only happened for the lack of documentation and clear coding.
> For instance, when I read the commit “2c4ea503f92bcf9c611f409d5cdecb”,
> it
> says it introduced an “optimization”, then I assumed that the process
> executed with our code base before commit “
> 2c4ea503f92bcf9c611f409d5cdecb”
> is working, but not as fast or with as much quality as the code with
> “2c4ea503f92bcf9c611f409d5cdecb”. However, that is not the case; the
> code
> “2c4ea503f92bcf9c611f409d5cdecb” is not optimizing anything; it is, in
> fact, fixing/creating a workflow to create templates from snapshots in
> XenServer deployments.
>
> The first PR(#1176) intended to solve #CLOUDSTACK-9025 was only
> tackling
> the problem for CloudStack deployments that use single hypervisor types
> (restricted to XenServers host in the same zone; this means, it was not
> expecting to have multiple hypervisors types in the same zone).
> Additionally, the lack of information regarding that solution
> (documentation, test cases and description in PRs and Jira ticket) led
> the
> code to be removed in #1124 after a long discussion and analysis in
> #1056.
> That piece of code seemed logicless.  It would receive a hostId and
> then
> change that hostId for other hostId of the zone without doing any
> check; it
> was not even checking the hypervisor and storage in which the host was
> connected to.
>
> The problem reported in #CLOUDSTACK-9025 is caused by partial snapshots
> that are taken in XenServer. This means, we do not take a complete
> snapshot, but a partial one that contains only the modified data. This
> requires rebuilding the VHD hierarchy when creating a template out of
> the
> snapshot. The point is that the first hostId received is not a hostId,
> but
> a system VM ID(SSVM). That is why the code in #1176 fixed the problem
> for
> 

Re: experienced Apache CloudStack Developer

[Rafael Weingärtner]

Hello Heinz,
Are you looking for a remote contractor? Or, would the candidate need to
relocate?

On Fri, Nov 10, 2017 at 5:26 AM,  wrote:

> Hi
>
> Do you know some skilled and experienced Apache CloudStack Developer, how
> likes to work for a successful Swiss Cloud Service Provider company?
> If so, please send the application to cl...@1st.ch. Of course the work
> would be contributed to the community (no forc).
> Thx
>
> Regards
> Heinz
>



-- 
Rafael Weingärtner

Re: HTTPS LB and x-forwarded-for

[Pierre-Luc Dion]

I kind of like the proxy backend type, ill check on our end if that would
work but definitely a simple and efficient approach!



Le 10 nov. 2017 01 h 44, "Wido den Hollander"  a écrit :

>
> > Op 9 november 2017 om 19:59 schreef Nux! :
> >
> >
> > Wido,
> >
> > Excellent suggestion with the "transparent proxy", I was not aware of
> that.
> > I think that would be a great idea and wouldn't require too many
> modifications, especially as Haproxy comes already with the VR.
> >
>
> It's indeed just a matter of a HAProxy config setting. We could make it
> configurable per backend in HAProxy. Regular HTTP, TCP or PROXY for example.
>
> That way your problem would be solved.
>
> Wido
>
> > To Paul:
> > - imho the LB solution ACS ships now is a bit handicaped since you do
> not know the remote host ip. You're flying blind unless you use google
> analytics (and these things have gotten more and more aggressively filtered
> by adblocks).
> > Enhancing Haproxy as Wido suggested would go a long way, it wouldn't
> break existing functionality and would also keep SSL processing off the VR.
> >
> > --
> > Sent from the Delta quadrant using Borg technology!
> >
> > Nux!
> > www.nux.ro
> >
> > - Original Message -
> > > From: "Andrija Panic" 
> > > To: "users" 
> > > Cc: "Khosrow Moossavi" , "Will Stevens" <
> wstev...@cloudops.com>, "dev"
> > > , "Pierre-Luc Dion" 
> > > Sent: Thursday, 9 November, 2017 13:10:58
> > > Subject: Re: HTTPS LB and x-forwarded-for
> >
> > > Wido,
> > >
> > > backend servers are not Linux only, for example we have a ton of
> Windows
> > > customers, some WEB solutions / IIS etc...
> > >
> > > @all - If we try to please/solve everyone's proxying
> solution/requirement -
> > > this is impossible IMHO - I'm thinking more about some "do it as you
> like"
> > > solution, to let customer write his own haproxy config and upoad it
> (for
> > > example, or something better?).
> > >
> > > We can support newer version of haproxy (1.5+) which also implement
> > > "transarent proxy" (integrate with kernel so to speak)  to allow
> TCP-level
> > > connections to backend (TCP mode, not HTTP mode) but to still
> "preserve"
> > > remote IP by faking it (fake soruce IP = transarent proxy).
> > >
> > > For the rest of configuration options,  I would leave it to the
> customer
> > > how he/she wants to configure rest of haproxy configuration, inlcuding
> > > custom checks, etc. Haproxy configuration is never-ending story, and we
> > > probably should allow custom sripts/configuration instead of trying to
> > > provide GUI/API way to configure everything (which is impossible...)
> > >
> > > Just my 2 cents...
> > >
> > > On 9 November 2017 at 08:13, Wido den Hollander 
> wrote:
> > >
> > >>
> > >> > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion <
> pd...@cloudops.com
> > >> >:
> > >> >
> > >> >
> > >> > Same challenge here too!
> > >> >
> > >> > Let's look at improving Load-balancing offering from cloudstack, I
> guest
> > >> we
> > >> > should do a feature spec draft soon..,  from my perspective, doing
> SSL
> > >> > offload on the VR could be problematic if the VR spec if too small,
> and
> > >> the
> > >> > default spec of the VR being 1vcpu@256MB, considering it can be the
> > >> router
> > >> > of a VPC, doing VPN termination, adding HTTPS  is a bit ish... What
> would
> > >> > be your thought about this ?
> > >> >
> > >> > I'd be curious to have a LB offering in ACS where it would deploy a
> > >> > redundant traefik[1] beside the VR for doing http and https
> > >> Load-balancing.
> > >> > I think it would also be useful if the API of that traefik instance
> would
> > >> > be available from within the VPC or LBnetwork so is API would be
> > >> accessible
> > >> > to other apps orchestration tools such as  kubernetes or rancher.
> > >> >
> > >> > traefik or not, here is what I think is needed by cloudstack in the
> LB
> > >> > improvement:
> > >> >
> > >> > - support http, https (X-Forwarded-For)
> > >>
> > >> HAProxy also supports the PROXY protocol towards the backends. Apache
> > >> 2.4.22 supports this natively and Varnish for example can also talk
> PROXY.
> > >>
> > >> It adds a littlebit of metadata to the connection so that the backend
> > >> knows the original IP the connection came from for example:
> > >> https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
> > >>
> > >> Wido
> > >>
> > >> > - basic persistence tuning (API already exist)
> > >> > - better backend monitoring, currently only a tcp connect validate
> if the
> > >> > webserver is up.
> > >> > - ssl offload
> > >> > - metric collection, more stats, maybe just export the tool status
> page
> > >> to
> > >> > the private network.
> > >> > - Container world support, right now if you have Rancher or
> kubernetes
> > >> > cluster, you need to deploy your own LB solution behing mostlikely a
> > >> static
> > >> > nat., If cloudstack would deploy a traefik instance, Kub or Rancher
> could
> > >> > reuse this instance and man

Re: HTTPS LB and x-forwarded-for

[Wido den Hollander]

> Op 10 november 2017 om 14:27 schreef Pierre-Luc Dion :
> 
> 
> I kind of like the proxy backend type, ill check on our end if that would
> work but definitely a simple and efficient approach!
> 

See: https://www.haproxy.com/blog/haproxy/proxy-protocol/

Apache HTTPd supports PROXY since 2.4.28: 
https://httpd.apache.org/docs/trunk/mod/mod_remoteip.html#remoteipproxyprotocol

"RemoteIPProxyProtocol is only available in httpd 2.4.28 and newer"

Wido

> 
> 
> Le 10 nov. 2017 01 h 44, "Wido den Hollander"  a écrit :
> 
> >
> > > Op 9 november 2017 om 19:59 schreef Nux! :
> > >
> > >
> > > Wido,
> > >
> > > Excellent suggestion with the "transparent proxy", I was not aware of
> > that.
> > > I think that would be a great idea and wouldn't require too many
> > modifications, especially as Haproxy comes already with the VR.
> > >
> >
> > It's indeed just a matter of a HAProxy config setting. We could make it
> > configurable per backend in HAProxy. Regular HTTP, TCP or PROXY for example.
> >
> > That way your problem would be solved.
> >
> > Wido
> >
> > > To Paul:
> > > - imho the LB solution ACS ships now is a bit handicaped since you do
> > not know the remote host ip. You're flying blind unless you use google
> > analytics (and these things have gotten more and more aggressively filtered
> > by adblocks).
> > > Enhancing Haproxy as Wido suggested would go a long way, it wouldn't
> > break existing functionality and would also keep SSL processing off the VR.
> > >
> > > --
> > > Sent from the Delta quadrant using Borg technology!
> > >
> > > Nux!
> > > www.nux.ro
> > >
> > > - Original Message -
> > > > From: "Andrija Panic" 
> > > > To: "users" 
> > > > Cc: "Khosrow Moossavi" , "Will Stevens" <
> > wstev...@cloudops.com>, "dev"
> > > > , "Pierre-Luc Dion" 
> > > > Sent: Thursday, 9 November, 2017 13:10:58
> > > > Subject: Re: HTTPS LB and x-forwarded-for
> > >
> > > > Wido,
> > > >
> > > > backend servers are not Linux only, for example we have a ton of
> > Windows
> > > > customers, some WEB solutions / IIS etc...
> > > >
> > > > @all - If we try to please/solve everyone's proxying
> > solution/requirement -
> > > > this is impossible IMHO - I'm thinking more about some "do it as you
> > like"
> > > > solution, to let customer write his own haproxy config and upoad it
> > (for
> > > > example, or something better?).
> > > >
> > > > We can support newer version of haproxy (1.5+) which also implement
> > > > "transarent proxy" (integrate with kernel so to speak)  to allow
> > TCP-level
> > > > connections to backend (TCP mode, not HTTP mode) but to still
> > "preserve"
> > > > remote IP by faking it (fake soruce IP = transarent proxy).
> > > >
> > > > For the rest of configuration options,  I would leave it to the
> > customer
> > > > how he/she wants to configure rest of haproxy configuration, inlcuding
> > > > custom checks, etc. Haproxy configuration is never-ending story, and we
> > > > probably should allow custom sripts/configuration instead of trying to
> > > > provide GUI/API way to configure everything (which is impossible...)
> > > >
> > > > Just my 2 cents...
> > > >
> > > > On 9 November 2017 at 08:13, Wido den Hollander 
> > wrote:
> > > >
> > > >>
> > > >> > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion <
> > pd...@cloudops.com
> > > >> >:
> > > >> >
> > > >> >
> > > >> > Same challenge here too!
> > > >> >
> > > >> > Let's look at improving Load-balancing offering from cloudstack, I
> > guest
> > > >> we
> > > >> > should do a feature spec draft soon..,  from my perspective, doing
> > SSL
> > > >> > offload on the VR could be problematic if the VR spec if too small,
> > and
> > > >> the
> > > >> > default spec of the VR being 1vcpu@256MB, considering it can be the
> > > >> router
> > > >> > of a VPC, doing VPN termination, adding HTTPS  is a bit ish... What
> > would
> > > >> > be your thought about this ?
> > > >> >
> > > >> > I'd be curious to have a LB offering in ACS where it would deploy a
> > > >> > redundant traefik[1] beside the VR for doing http and https
> > > >> Load-balancing.
> > > >> > I think it would also be useful if the API of that traefik instance
> > would
> > > >> > be available from within the VPC or LBnetwork so is API would be
> > > >> accessible
> > > >> > to other apps orchestration tools such as  kubernetes or rancher.
> > > >> >
> > > >> > traefik or not, here is what I think is needed by cloudstack in the
> > LB
> > > >> > improvement:
> > > >> >
> > > >> > - support http, https (X-Forwarded-For)
> > > >>
> > > >> HAProxy also supports the PROXY protocol towards the backends. Apache
> > > >> 2.4.22 supports this natively and Varnish for example can also talk
> > PROXY.
> > > >>
> > > >> It adds a littlebit of metadata to the connection so that the backend
> > > >> knows the original IP the connection came from for example:
> > > >> https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt
> > > >>
> > > >> Wido
> > > >>
> > > >> > - basic persiste

Re: experienced Apache CloudStack Developer

[cloud]

Hi Rafael

We currently try to figure out all options we have. Of course it would be most 
efficient to have people on our premise, but we are open for other ways of 
collaboration.

regards
Heinz

- Original Message -
From: "Rafael Weingärtner" 
To: "dev" 
Sent: Friday, November 10, 2017 1:18:49 PM
Subject: Re: experienced Apache CloudStack Developer

Hello Heinz,
Are you looking for a remote contractor? Or, would the candidate need to
relocate?

On Fri, Nov 10, 2017 at 5:26 AM,  wrote:

> Hi
>
> Do you know some skilled and experienced Apache CloudStack Developer, how
> likes to work for a successful Swiss Cloud Service Provider company?
> If so, please send the application to cl...@1st.ch. Of course the work
> would be contributed to the community (no forc).
> Thx
>
> Regards
> Heinz
>



-- 
Rafael Weingärtner

Re: experienced Apache CloudStack Developer

[Rafael Weingärtner]

Thanks for the details; I think this information is useful for candidates.

I also recall reading something about a page to post Apache related jobs; I
think it happened at d...@community.apache.org (or some cloudstack list?),
but I am not finding it anymore.

On Fri, Nov 10, 2017 at 12:55 PM,  wrote:

> Hi Rafael
>
> We currently try to figure out all options we have. Of course it would be
> most efficient to have people on our premise, but we are open for other
> ways of collaboration.
>
> regards
> Heinz
>
> - Original Message -
> From: "Rafael Weingärtner" 
> To: "dev" 
> Sent: Friday, November 10, 2017 1:18:49 PM
> Subject: Re: experienced Apache CloudStack Developer
>
> Hello Heinz,
> Are you looking for a remote contractor? Or, would the candidate need to
> relocate?
>
> On Fri, Nov 10, 2017 at 5:26 AM,  wrote:
>
> > Hi
> >
> > Do you know some skilled and experienced Apache CloudStack Developer, how
> > likes to work for a successful Swiss Cloud Service Provider company?
> > If so, please send the application to cl...@1st.ch. Of course the work
> > would be contributed to the community (no forc).
> > Thx
> >
> > Regards
> > Heinz
> >
>
>
>
> --
> Rafael Weingärtner
>



-- 
Rafael Weingärtner

Re: HTTPS LB and x-forwarded-for

[Pierre-Luc Dion]

Hi Wido, do you know if this would work for https traffic too?

Le 10 nov. 2017 09 h 35, "Wido den Hollander"  a écrit :

>
> > Op 10 november 2017 om 14:27 schreef Pierre-Luc Dion  >:
> >
> >
> > I kind of like the proxy backend type, ill check on our end if that would
> > work but definitely a simple and efficient approach!
> >
>
> See: https://www.haproxy.com/blog/haproxy/proxy-protocol/
>
> Apache HTTPd supports PROXY since 2.4.28: https://httpd.apache.org/docs/
> trunk/mod/mod_remoteip.html#remoteipproxyprotocol
>
> "RemoteIPProxyProtocol is only available in httpd 2.4.28 and newer"
>
> Wido
>
> >
> >
> > Le 10 nov. 2017 01 h 44, "Wido den Hollander"  a écrit :
> >
> > >
> > > > Op 9 november 2017 om 19:59 schreef Nux! :
> > > >
> > > >
> > > > Wido,
> > > >
> > > > Excellent suggestion with the "transparent proxy", I was not aware of
> > > that.
> > > > I think that would be a great idea and wouldn't require too many
> > > modifications, especially as Haproxy comes already with the VR.
> > > >
> > >
> > > It's indeed just a matter of a HAProxy config setting. We could make it
> > > configurable per backend in HAProxy. Regular HTTP, TCP or PROXY for
> example.
> > >
> > > That way your problem would be solved.
> > >
> > > Wido
> > >
> > > > To Paul:
> > > > - imho the LB solution ACS ships now is a bit handicaped since you do
> > > not know the remote host ip. You're flying blind unless you use google
> > > analytics (and these things have gotten more and more aggressively
> filtered
> > > by adblocks).
> > > > Enhancing Haproxy as Wido suggested would go a long way, it wouldn't
> > > break existing functionality and would also keep SSL processing off
> the VR.
> > > >
> > > > --
> > > > Sent from the Delta quadrant using Borg technology!
> > > >
> > > > Nux!
> > > > www.nux.ro
> > > >
> > > > - Original Message -
> > > > > From: "Andrija Panic" 
> > > > > To: "users" 
> > > > > Cc: "Khosrow Moossavi" , "Will Stevens" <
> > > wstev...@cloudops.com>, "dev"
> > > > > , "Pierre-Luc Dion"  >
> > > > > Sent: Thursday, 9 November, 2017 13:10:58
> > > > > Subject: Re: HTTPS LB and x-forwarded-for
> > > >
> > > > > Wido,
> > > > >
> > > > > backend servers are not Linux only, for example we have a ton of
> > > Windows
> > > > > customers, some WEB solutions / IIS etc...
> > > > >
> > > > > @all - If we try to please/solve everyone's proxying
> > > solution/requirement -
> > > > > this is impossible IMHO - I'm thinking more about some "do it as
> you
> > > like"
> > > > > solution, to let customer write his own haproxy config and upoad it
> > > (for
> > > > > example, or something better?).
> > > > >
> > > > > We can support newer version of haproxy (1.5+) which also implement
> > > > > "transarent proxy" (integrate with kernel so to speak)  to allow
> > > TCP-level
> > > > > connections to backend (TCP mode, not HTTP mode) but to still
> > > "preserve"
> > > > > remote IP by faking it (fake soruce IP = transarent proxy).
> > > > >
> > > > > For the rest of configuration options,  I would leave it to the
> > > customer
> > > > > how he/she wants to configure rest of haproxy configuration,
> inlcuding
> > > > > custom checks, etc. Haproxy configuration is never-ending story,
> and we
> > > > > probably should allow custom sripts/configuration instead of
> trying to
> > > > > provide GUI/API way to configure everything (which is
> impossible...)
> > > > >
> > > > > Just my 2 cents...
> > > > >
> > > > > On 9 November 2017 at 08:13, Wido den Hollander 
> > > wrote:
> > > > >
> > > > >>
> > > > >> > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion <
> > > pd...@cloudops.com
> > > > >> >:
> > > > >> >
> > > > >> >
> > > > >> > Same challenge here too!
> > > > >> >
> > > > >> > Let's look at improving Load-balancing offering from
> cloudstack, I
> > > guest
> > > > >> we
> > > > >> > should do a feature spec draft soon..,  from my perspective,
> doing
> > > SSL
> > > > >> > offload on the VR could be problematic if the VR spec if too
> small,
> > > and
> > > > >> the
> > > > >> > default spec of the VR being 1vcpu@256MB, considering it can
> be the
> > > > >> router
> > > > >> > of a VPC, doing VPN termination, adding HTTPS  is a bit ish...
> What
> > > would
> > > > >> > be your thought about this ?
> > > > >> >
> > > > >> > I'd be curious to have a LB offering in ACS where it would
> deploy a
> > > > >> > redundant traefik[1] beside the VR for doing http and https
> > > > >> Load-balancing.
> > > > >> > I think it would also be useful if the API of that traefik
> instance
> > > would
> > > > >> > be available from within the VPC or LBnetwork so is API would be
> > > > >> accessible
> > > > >> > to other apps orchestration tools such as  kubernetes or
> rancher.
> > > > >> >
> > > > >> > traefik or not, here is what I think is needed by cloudstack in
> the
> > > LB
> > > > >> > improvement:
> > > > >> >
> > > > >> > - support http, https (X-Forwarded-For)
> > > > >>
> > > > >> HAProxy also supports the P

Re: HTTPS LB and x-forwarded-for

[Nux!]

Pierre-Luc,

Haproxy docs say it should work for any kind of traffic as long as both ends 
are PROXY-aware and it look like a majority of software is.
So, in short, yes.

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

- Original Message -
> From: "Pierre-Luc Dion" 
> To: "Wido den Hollander" 
> Cc: "dev" , "Khosrow Moossavi" 
> , "Will Stevens"
> , "Nux!" , "users" 
> 
> Sent: Friday, 10 November, 2017 15:32:38
> Subject: Re: HTTPS LB and x-forwarded-for

> Hi Wido, do you know if this would work for https traffic too?
> 
> Le 10 nov. 2017 09 h 35, "Wido den Hollander"  a écrit :
> 
>>
>> > Op 10 november 2017 om 14:27 schreef Pierre-Luc Dion > >:
>> >
>> >
>> > I kind of like the proxy backend type, ill check on our end if that would
>> > work but definitely a simple and efficient approach!
>> >
>>
>> See: https://www.haproxy.com/blog/haproxy/proxy-protocol/
>>
>> Apache HTTPd supports PROXY since 2.4.28: https://httpd.apache.org/docs/
>> trunk/mod/mod_remoteip.html#remoteipproxyprotocol
>>
>> "RemoteIPProxyProtocol is only available in httpd 2.4.28 and newer"
>>
>> Wido
>>
>> >
>> >
>> > Le 10 nov. 2017 01 h 44, "Wido den Hollander"  a écrit :
>> >
>> > >
>> > > > Op 9 november 2017 om 19:59 schreef Nux! :
>> > > >
>> > > >
>> > > > Wido,
>> > > >
>> > > > Excellent suggestion with the "transparent proxy", I was not aware of
>> > > that.
>> > > > I think that would be a great idea and wouldn't require too many
>> > > modifications, especially as Haproxy comes already with the VR.
>> > > >
>> > >
>> > > It's indeed just a matter of a HAProxy config setting. We could make it
>> > > configurable per backend in HAProxy. Regular HTTP, TCP or PROXY for
>> example.
>> > >
>> > > That way your problem would be solved.
>> > >
>> > > Wido
>> > >
>> > > > To Paul:
>> > > > - imho the LB solution ACS ships now is a bit handicaped since you do
>> > > not know the remote host ip. You're flying blind unless you use google
>> > > analytics (and these things have gotten more and more aggressively
>> filtered
>> > > by adblocks).
>> > > > Enhancing Haproxy as Wido suggested would go a long way, it wouldn't
>> > > break existing functionality and would also keep SSL processing off
>> the VR.
>> > > >
>> > > > --
>> > > > Sent from the Delta quadrant using Borg technology!
>> > > >
>> > > > Nux!
>> > > > www.nux.ro
>> > > >
>> > > > - Original Message -
>> > > > > From: "Andrija Panic" 
>> > > > > To: "users" 
>> > > > > Cc: "Khosrow Moossavi" , "Will Stevens" <
>> > > wstev...@cloudops.com>, "dev"
>> > > > > , "Pierre-Luc Dion" > >
>> > > > > Sent: Thursday, 9 November, 2017 13:10:58
>> > > > > Subject: Re: HTTPS LB and x-forwarded-for
>> > > >
>> > > > > Wido,
>> > > > >
>> > > > > backend servers are not Linux only, for example we have a ton of
>> > > Windows
>> > > > > customers, some WEB solutions / IIS etc...
>> > > > >
>> > > > > @all - If we try to please/solve everyone's proxying
>> > > solution/requirement -
>> > > > > this is impossible IMHO - I'm thinking more about some "do it as
>> you
>> > > like"
>> > > > > solution, to let customer write his own haproxy config and upoad it
>> > > (for
>> > > > > example, or something better?).
>> > > > >
>> > > > > We can support newer version of haproxy (1.5+) which also implement
>> > > > > "transarent proxy" (integrate with kernel so to speak)  to allow
>> > > TCP-level
>> > > > > connections to backend (TCP mode, not HTTP mode) but to still
>> > > "preserve"
>> > > > > remote IP by faking it (fake soruce IP = transarent proxy).
>> > > > >
>> > > > > For the rest of configuration options,  I would leave it to the
>> > > customer
>> > > > > how he/she wants to configure rest of haproxy configuration,
>> inlcuding
>> > > > > custom checks, etc. Haproxy configuration is never-ending story,
>> and we
>> > > > > probably should allow custom sripts/configuration instead of
>> trying to
>> > > > > provide GUI/API way to configure everything (which is
>> impossible...)
>> > > > >
>> > > > > Just my 2 cents...
>> > > > >
>> > > > > On 9 November 2017 at 08:13, Wido den Hollander 
>> > > wrote:
>> > > > >
>> > > > >>
>> > > > >> > Op 8 november 2017 om 14:59 schreef Pierre-Luc Dion <
>> > > pd...@cloudops.com
>> > > > >> >:
>> > > > >> >
>> > > > >> >
>> > > > >> > Same challenge here too!
>> > > > >> >
>> > > > >> > Let's look at improving Load-balancing offering from
>> cloudstack, I
>> > > guest
>> > > > >> we
>> > > > >> > should do a feature spec draft soon..,  from my perspective,
>> doing
>> > > SSL
>> > > > >> > offload on the VR could be problematic if the VR spec if too
>> small,
>> > > and
>> > > > >> the
>> > > > >> > default spec of the VR being 1vcpu@256MB, considering it can
>> be the
>> > > > >> router
>> > > > >> > of a VPC, doing VPN termination, adding HTTPS  is a bit ish...
>> What
>> > > would
>> > > > >> > be your thought about this ?
>> > > > >> >
>> > > > >> > I'd be curious to have a LB offering in ACS where it wou

Apache CloudStack 4.10 VR/BasicZone/KVM Problem

[Ivan Kudryavtsev]

Hello, Devs.

I experience VR Start Problem in the fresh ACS 4.10 deployment

Intersting place of logs is here: https://pastebin.com/iBXRBA5N

Basically, the situation looks like:

1. Management Server tries to launch VR
2. It gets from Agent proper VR response with VR details
3. It sends StopCommand without explanation.

I'm trying to figure out what happens inside, but the codebase is huge and
still no positive results. Please, let me know if you have any ideas which
could help me finding the reason. Thanks a lot.

-- 
With best regards, Ivan Kudryavtsev
Bitworks Software, Ltd.
Cell: +7-923-414-1515
WWW: http://bitworks.software/ 

Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

[Simon Weller]

Ivan,


Can you put the host agents into debug mode? Hopefully that will provide more 
information.


https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug


- Si


From: Ivan Kudryavtsev 
Sent: Friday, November 10, 2017 11:34 AM
To: dev@cloudstack.apache.org
Subject: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

Hello, Devs.

I experience VR Start Problem in the fresh ACS 4.10 deployment

Intersting place of logs is here: https://pastebin.com/iBXRBA5N

Basically, the situation looks like:

1. Management Server tries to launch VR
2. It gets from Agent proper VR response with VR details
3. It sends StopCommand without explanation.

I'm trying to figure out what happens inside, but the codebase is huge and
still no positive results. Please, let me know if you have any ideas which
could help me finding the reason. Thanks a lot.

--
With best regards, Ivan Kudryavtsev
Bitworks Software, Ltd.
Cell: +7-923-414-1515
WWW: http://bitworks.software/ 

Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

[Ivan Kudryavtsev]

Hi, I did, and it does the things right, I even added "tee" to ssh 3922
communication script to out vr response to additional log and it only
receives VR version line and sends all info (the same from pastebin) to ACS
and receives "stop" order.

I'll try to provide additional info, but ad you can see, management
receives proper response and sends stop next op. It looks very freaky
without any notification...

11 нояб. 2017 г. 0:37 пользователь "Simon Weller" 
написал:

> Ivan,
>
>
> Can you put the host agents into debug mode? Hopefully that will provide
> more information.
>
>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug
>
>
> - Si
>
> 
> From: Ivan Kudryavtsev 
> Sent: Friday, November 10, 2017 11:34 AM
> To: dev@cloudstack.apache.org
> Subject: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
>
> Hello, Devs.
>
> I experience VR Start Problem in the fresh ACS 4.10 deployment
>
> Intersting place of logs is here: https://pastebin.com/iBXRBA5N
>
> Basically, the situation looks like:
>
> 1. Management Server tries to launch VR
> 2. It gets from Agent proper VR response with VR details
> 3. It sends StopCommand without explanation.
>
> I'm trying to figure out what happens inside, but the codebase is huge and
> still no positive results. Please, let me know if you have any ideas which
> could help me finding the reason. Thanks a lot.
>
> --
> With best regards, Ivan Kudryavtsev
> Bitworks Software, Ltd.
> Cell: +7-923-414-1515
> WWW: http://bitworks.software/ 
>

Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

[Simon Weller]

Is the storage ceph?



From: Ivan Kudryavtsev 
Sent: Friday, November 10, 2017 11:52 AM
To: dev@cloudstack.apache.org
Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

Hi, I did, and it does the things right, I even added "tee" to ssh 3922
communication script to out vr response to additional log and it only
receives VR version line and sends all info (the same from pastebin) to ACS
and receives "stop" order.

I'll try to provide additional info, but ad you can see, management
receives proper response and sends stop next op. It looks very freaky
without any notification...

11 нояб. 2017 г. 0:37 пользователь "Simon Weller" 
написал:

> Ivan,
>
>
> Can you put the host agents into debug mode? Hopefully that will provide
> more information.
>
>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug
>
>
> - Si
>
> 
> From: Ivan Kudryavtsev 
> Sent: Friday, November 10, 2017 11:34 AM
> To: dev@cloudstack.apache.org
> Subject: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
>
> Hello, Devs.
>
> I experience VR Start Problem in the fresh ACS 4.10 deployment
>
> Intersting place of logs is here: https://pastebin.com/iBXRBA5N
>
> Basically, the situation looks like:
>
> 1. Management Server tries to launch VR
> 2. It gets from Agent proper VR response with VR details
> 3. It sends StopCommand without explanation.
>
> I'm trying to figure out what happens inside, but the codebase is huge and
> still no positive results. Please, let me know if you have any ideas which
> could help me finding the reason. Thanks a lot.
>
> --
> With best regards, Ivan Kudryavtsev
> Bitworks Software, Ltd.
> Cell: +7-923-414-1515
> WWW: http://bitworks.software/ 
>

Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

[Ivan Kudryavtsev]

Hi. No, regular NFS. VR starts great, but stopped by ms, other system vms
are working. I even added to communication script on compute node "sleep
3600" before ssh, so response to management is delayed, I logged so to VR,
all interfaces are up, iptables rules are OK.

So agent rolls vr good, but stops it by management order with no obvious
reason.

11 нояб. 2017 г. 0:54 пользователь "Simon Weller" 
написал:

> Is the storage ceph?
>
>
> 
> From: Ivan Kudryavtsev 
> Sent: Friday, November 10, 2017 11:52 AM
> To: dev@cloudstack.apache.org
> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
>
> Hi, I did, and it does the things right, I even added "tee" to ssh 3922
> communication script to out vr response to additional log and it only
> receives VR version line and sends all info (the same from pastebin) to ACS
> and receives "stop" order.
>
> I'll try to provide additional info, but ad you can see, management
> receives proper response and sends stop next op. It looks very freaky
> without any notification...
>
> 11 нояб. 2017 г. 0:37 пользователь "Simon Weller"  >
> написал:
>
> > Ivan,
> >
> >
> > Can you put the host agents into debug mode? Hopefully that will provide
> > more information.
> >
> >
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug
> >
> >
> > - Si
> >
> > 
> > From: Ivan Kudryavtsev 
> > Sent: Friday, November 10, 2017 11:34 AM
> > To: dev@cloudstack.apache.org
> > Subject: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> >
> > Hello, Devs.
> >
> > I experience VR Start Problem in the fresh ACS 4.10 deployment
> >
> > Intersting place of logs is here: https://pastebin.com/iBXRBA5N
> >
> > Basically, the situation looks like:
> >
> > 1. Management Server tries to launch VR
> > 2. It gets from Agent proper VR response with VR details
> > 3. It sends StopCommand without explanation.
> >
> > I'm trying to figure out what happens inside, but the codebase is huge
> and
> > still no positive results. Please, let me know if you have any ideas
> which
> > could help me finding the reason. Thanks a lot.
> >
> > --
> > With best regards, Ivan Kudryavtsev
> > Bitworks Software, Ltd.
> > Cell: +7-923-414-1515
> > WWW: http://bitworks.software/ 
> >
>

Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

[Simon Weller]

What VR template image are you using?



From: Ivan Kudryavtsev 
Sent: Friday, November 10, 2017 11:59 AM
To: dev@cloudstack.apache.org
Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

Hi. No, regular NFS. VR starts great, but stopped by ms, other system vms
are working. I even added to communication script on compute node "sleep
3600" before ssh, so response to management is delayed, I logged so to VR,
all interfaces are up, iptables rules are OK.

So agent rolls vr good, but stops it by management order with no obvious
reason.

11 нояб. 2017 г. 0:54 пользователь "Simon Weller" 
написал:

> Is the storage ceph?
>
>
> 
> From: Ivan Kudryavtsev 
> Sent: Friday, November 10, 2017 11:52 AM
> To: dev@cloudstack.apache.org
> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
>
> Hi, I did, and it does the things right, I even added "tee" to ssh 3922
> communication script to out vr response to additional log and it only
> receives VR version line and sends all info (the same from pastebin) to ACS
> and receives "stop" order.
>
> I'll try to provide additional info, but ad you can see, management
> receives proper response and sends stop next op. It looks very freaky
> without any notification...
>
> 11 нояб. 2017 г. 0:37 пользователь "Simon Weller"  >
> написал:
>
> > Ivan,
> >
> >
> > Can you put the host agents into debug mode? Hopefully that will provide
> > more information.
> >
> >
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+debug
KVM agent debug - Apache Cloudstack - Apache Software 
...
cwiki.apache.org
Steps to debug the KVM agent from eclipse: In KVM agent edit 
‘/usr/libexec/agent-runner ‘, add "-Xrunjdwp:transport=dt_socket,address=8787 
...



> >
> >
> > - Si
> >
> > 
> > From: Ivan Kudryavtsev 
> > Sent: Friday, November 10, 2017 11:34 AM
> > To: dev@cloudstack.apache.org
> > Subject: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> >
> > Hello, Devs.
> >
> > I experience VR Start Problem in the fresh ACS 4.10 deployment
> >
> > Intersting place of logs is here: https://pastebin.com/iBXRBA5N
[https://pastebin.com/i/facebook.png]

2017-11-10 23:05:35,853 DEBUG [c.c.a.t.Request] 
(Work-Job-Executor-15:ctx-6fdf61 - Pastebin.com
pastebin.com



> >
> > Basically, the situation looks like:
> >
> > 1. Management Server tries to launch VR
> > 2. It gets from Agent proper VR response with VR details
> > 3. It sends StopCommand without explanation.
> >
> > I'm trying to figure out what happens inside, but the codebase is huge
> and
> > still no positive results. Please, let me know if you have any ideas
> which
> > could help me finding the reason. Thanks a lot.
> >
> > --
> > With best regards, Ivan Kudryavtsev
> > Bitworks Software, Ltd.
> > Cell: +7-923-414-1515
> > WWW: http://bitworks.software/ 
> >
>

RE: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

[Paul Angus]

Ivan,

Can you paste a larger section of unfiltered logs.  There would always be a 
message explaining why the mgmt. server thought that a VR should be shut down 



Kind regards,

Paul Angus

paul.an...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 


-Original Message-
From: Simon Weller [mailto:swel...@ena.com.INVALID] 
Sent: 10 November 2017 18:39
To: dev@cloudstack.apache.org
Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

What VR template image are you using?



From: Ivan Kudryavtsev 
Sent: Friday, November 10, 2017 11:59 AM
To: dev@cloudstack.apache.org
Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem

Hi. No, regular NFS. VR starts great, but stopped by ms, other system vms are 
working. I even added to communication script on compute node "sleep 3600" 
before ssh, so response to management is delayed, I logged so to VR, all 
interfaces are up, iptables rules are OK.

So agent rolls vr good, but stops it by management order with no obvious reason.

11 нояб. 2017 г. 0:54 пользователь "Simon Weller" 
написал:

> Is the storage ceph?
>
>
> 
> From: Ivan Kudryavtsev 
> Sent: Friday, November 10, 2017 11:52 AM
> To: dev@cloudstack.apache.org
> Subject: Re: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
>
> Hi, I did, and it does the things right, I even added "tee" to ssh 
> 3922 communication script to out vr response to additional log and it 
> only receives VR version line and sends all info (the same from 
> pastebin) to ACS and receives "stop" order.
>
> I'll try to provide additional info, but ad you can see, management 
> receives proper response and sends stop next op. It looks very freaky 
> without any notification...
>
> 11 нояб. 2017 г. 0:37 пользователь "Simon Weller" 
>  >
> написал:
>
> > Ivan,
> >
> >
> > Can you put the host agents into debug mode? Hopefully that will 
> > provide more information.
> >
> >
> > https://cwiki.apache.org/confluence/display/CLOUDSTACK/KVM+agent+deb
> > ug
KVM agent debug - Apache Cloudstack - Apache Software 
...
cwiki.apache.org
Steps to debug the KVM agent from eclipse: In KVM agent edit 
'/usr/libexec/agent-runner ', add "-Xrunjdwp:transport=dt_socket,address=8787 
...



> >
> >
> > - Si
> >
> > 
> > From: Ivan Kudryavtsev 
> > Sent: Friday, November 10, 2017 11:34 AM
> > To: dev@cloudstack.apache.org
> > Subject: Apache CloudStack 4.10 VR/BasicZone/KVM Problem
> >
> > Hello, Devs.
> >
> > I experience VR Start Problem in the fresh ACS 4.10 deployment
> >
> > Intersting place of logs is here: https://pastebin.com/iBXRBA5N
[https://pastebin.com/i/facebook.png]

2017-11-10 23:05:35,853 DEBUG [c.c.a.t.Request] 
(Work-Job-Executor-15:ctx-6fdf61 - Pastebin.com
pastebin.com



> >
> > Basically, the situation looks like:
> >
> > 1. Management Server tries to launch VR 2. It gets from Agent proper 
> > VR response with VR details 3. It sends StopCommand without 
> > explanation.
> >
> > I'm trying to figure out what happens inside, but the codebase is 
> > huge
> and
> > still no positive results. Please, let me know if you have any ideas
> which
> > could help me finding the reason. Thanks a lot.
> >
> > --
> > With best regards, Ivan Kudryavtsev
> > Bitworks Software, Ltd.
> > Cell: +7-923-414-1515
> > WWW: http://bitworks.software/ 
> >
>