Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-17 Thread Stefano Stabellini
On Tue, 9 Jun 2015, Ian Campbell wrote: > On Tue, 2015-06-09 at 11:14 +0100, Stefano Stabellini wrote: > > I don't think that the current solution is inherently racy. If we are > > really interested in an honest evaluation of the current solution in > > terms of races, I would be happy to do so. >

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-16 Thread Stefano Stabellini
On Tue, 9 Jun 2015, George Dunlap wrote: > On 06/09/2015 11:14 AM, Stefano Stabellini wrote: > > On Mon, 8 Jun 2015, George Dunlap wrote: > >> I think that qemu needs to tell libxl how much memory it is using for > >> all of its needs -- including option ROMs. (See my example below.) For > >> old

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-09 Thread Ian Campbell
On Tue, 2015-06-09 at 11:14 +0100, Stefano Stabellini wrote: > I don't think that the current solution is inherently racy. If we are > really interested in an honest evaluation of the current solution in > terms of races, I would be happy to do so. Consider a domain with 1M of RAM (==target and ma

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-09 Thread George Dunlap
On 06/09/2015 11:14 AM, Stefano Stabellini wrote: > On Mon, 8 Jun 2015, George Dunlap wrote: >> I think that qemu needs to tell libxl how much memory it is using for >> all of its needs -- including option ROMs. (See my example below.) For >> older qemus we can just make some assumptions like we

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-09 Thread Wei Liu
On Tue, Jun 09, 2015 at 11:00:12AM +0100, George Dunlap wrote: > >> I think that qemu needs to tell libxl how much memory it is using for > >> all of its needs -- including option ROMs. (See my example below.) For > >> older qemus we can just make some assumptions like we always have. > >> > > >

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-09 Thread Stefano Stabellini
On Mon, 8 Jun 2015, George Dunlap wrote: > I think that qemu needs to tell libxl how much memory it is using for > all of its needs -- including option ROMs. (See my example below.) For > older qemus we can just make some assumptions like we always have. I'll just note that I have still not seen

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-09 Thread George Dunlap
On 06/08/2015 05:06 PM, Don Slutz wrote: > On 06/08/15 11:37, George Dunlap wrote: We already have the problem that the balloon driver at the moment doesn't actually know how big the guest RAM is, nor , but is being told to make a balloon exactly big enough to bring the total RAM do

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Don Slutz
On 06/08/15 11:37, George Dunlap wrote: > On 06/08/2015 04:01 PM, Don Slutz wrote: >> On 06/08/15 10:20, George Dunlap wrote: >>> And at the moment, pages in the p2m are allocated by a number of entities: >>> * In the libxc domain builder. >>> * In the guest balloon driver >>> * And now, in qemu, t

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Konrad Rzeszutek Wilk
On Mon, Jun 08, 2015 at 04:20:48PM +0100, George Dunlap wrote: > On 06/08/2015 03:53 PM, Konrad Rzeszutek Wilk wrote: > > On Mon, Jun 08, 2015 at 01:11:38PM +0100, Ian Campbell wrote: > >> On Mon, 2015-06-08 at 12:40 +0100, Stefano Stabellini wrote: > >> > > I disagree that libxl should be the

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread George Dunlap
On 06/08/2015 04:01 PM, Don Slutz wrote: > On 06/08/15 10:20, George Dunlap wrote: >> And at the moment, pages in the p2m are allocated by a number of entities: >> * In the libxc domain builder. >> * In the guest balloon driver >> * And now, in qemu, to allocate extra memory for virtual ROMs. > >

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread George Dunlap
On 06/08/2015 03:53 PM, Konrad Rzeszutek Wilk wrote: > On Mon, Jun 08, 2015 at 01:11:38PM +0100, Ian Campbell wrote: >> On Mon, 2015-06-08 at 12:40 +0100, Stefano Stabellini wrote: >> > I disagree that libxl should be the arbitrator of a property that is > stored, maintained and enforced by

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Don Slutz
On 06/08/15 10:20, George Dunlap wrote: > On 06/08/2015 02:22 PM, Stefano Stabellini wrote: >>> 3. A group of entities which operate in isolation by only ever >>> increasing or descreasing the max pages according to their own >>> requirements, without reference to anyone else.

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Konrad Rzeszutek Wilk
On Mon, Jun 08, 2015 at 01:11:38PM +0100, Ian Campbell wrote: > On Mon, 2015-06-08 at 12:40 +0100, Stefano Stabellini wrote: > > > > > I disagree that libxl should be the arbitrator of a property that is > > > > stored, maintained and enforced by Xen. Xen should be the arbitrator. > > > > > > Tha

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread George Dunlap
On 06/08/2015 02:22 PM, Stefano Stabellini wrote: >> 3. A group of entities which operate in isolation by only ever >> increasing or descreasing the max pages according to their own >> requirements, without reference to anyone else. When QEMU >> entered the fray, and wi

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread George Dunlap
On 06/05/2015 06:10 PM, Stefano Stabellini wrote: >> 5. Add a user configurable field in current libxl JSON structure to >>record how much more memory this domain needs. Admin is required to >>fill in that value manually. In the mean time we revert the change in >>QEMU and declare QEMU

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Ian Campbell
On Mon, 2015-06-08 at 14:22 +0100, Stefano Stabellini wrote: > On Mon, 8 Jun 2015, Ian Campbell wrote: > > On Mon, 2015-06-08 at 12:40 +0100, Stefano Stabellini wrote: > > > > > > > I disagree that libxl should be the arbitrator of a property that is > > > > > stored, maintained and enforced by Xe

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Stefano Stabellini
On Mon, 8 Jun 2015, Andrew Cooper wrote: > On 08/06/15 14:38, Stefano Stabellini wrote: > >> Also device-mode/$domid/state is writable by QEMU so we can't trust > >> > the content as indicator either. > > We can because the write happens before we unpause the guest > > Only when creating the domai

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Andrew Cooper
On 08/06/15 14:38, Stefano Stabellini wrote: >> Also device-mode/$domid/state is writable by QEMU so we can't trust >> > the content as indicator either. > We can because the write happens before we unpause the guest Only when creating the domain fresh. On resume, the guest has possibly had the c

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Stefano Stabellini
On Mon, 8 Jun 2015, Wei Liu wrote: > On Mon, Jun 08, 2015 at 02:27:11PM +0100, Stefano Stabellini wrote: > > On Mon, 8 Jun 2015, Wei Liu wrote: > > > On Mon, Jun 08, 2015 at 12:39:52PM +0100, Stefano Stabellini wrote: > > > [...] > > > > > > > 3. Add a libxl layer that wraps necessary information,

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Jan Beulich
>>> On 08.06.15 at 15:01, wrote: > On Mon, 8 Jun 2015, Andrew Cooper wrote: >> At the moment, set_max_mem is the only method the toolstack has of >> putting a hard upper bound on a domains memory usage (along with a few >> others like the shadow mem size, and PoD cache.) >> >> In a disaggregated

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Wei Liu
On Mon, Jun 08, 2015 at 02:27:11PM +0100, Stefano Stabellini wrote: > On Mon, 8 Jun 2015, Wei Liu wrote: > > On Mon, Jun 08, 2015 at 12:39:52PM +0100, Stefano Stabellini wrote: > > [...] > > > > > > 3. Add a libxl layer that wraps necessary information, take over > > > > > >Andrew's work on lib

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Stefano Stabellini
On Mon, 8 Jun 2015, Wei Liu wrote: > On Mon, Jun 08, 2015 at 12:39:52PM +0100, Stefano Stabellini wrote: > [...] > > > > > 3. Add a libxl layer that wraps necessary information, take over > > > > >Andrew's work on libxl migration v2. Having a libxl layer that's > > > > > not > > > > >part

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Stefano Stabellini
On Mon, 8 Jun 2015, Ian Campbell wrote: > On Mon, 2015-06-08 at 12:40 +0100, Stefano Stabellini wrote: > > > > > I disagree that libxl should be the arbitrator of a property that is > > > > stored, maintained and enforced by Xen. Xen should be the arbitrator. > > > > > > That's not what "arbitrat

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Wei Liu
On Mon, Jun 08, 2015 at 12:39:52PM +0100, Stefano Stabellini wrote: [...] > > > > 3. Add a libxl layer that wraps necessary information, take over > > > >Andrew's work on libxl migration v2. Having a libxl layer that's not > > > >part of migration v2 is a waste of effort. > > > > > > > >

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Stefano Stabellini
On Mon, 8 Jun 2015, Andrew Cooper wrote: > On 08/06/15 12:39, Stefano Stabellini wrote: > > On Fri, 5 Jun 2015, Wei Liu wrote: > >> On Fri, Jun 05, 2015 at 06:10:17PM +0100, Stefano Stabellini wrote: > >>> On Fri, 5 Jun 2015, Wei Liu wrote: > Hi all > > This bug is now considered a b

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Andrew Cooper
On 08/06/15 12:39, Stefano Stabellini wrote: > On Fri, 5 Jun 2015, Wei Liu wrote: >> On Fri, Jun 05, 2015 at 06:10:17PM +0100, Stefano Stabellini wrote: >>> On Fri, 5 Jun 2015, Wei Liu wrote: Hi all This bug is now considered a blocker for 4.6 release. The premises of the p

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Ian Campbell
On Mon, 2015-06-08 at 12:40 +0100, Stefano Stabellini wrote: > > > I disagree that libxl should be the arbitrator of a property that is > > > stored, maintained and enforced by Xen. Xen should be the arbitrator. > > > > That's not what "arbitrator" means, an arbitrator decides what the value > >

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Stefano Stabellini
On Fri, 5 Jun 2015, Ian Campbell wrote: > On Fri, 2015-06-05 at 18:10 +0100, Stefano Stabellini wrote: > > On Fri, 5 Jun 2015, Wei Liu wrote: > > > Hi all > > > > > > This bug is now considered a blocker for 4.6 release. > > > > > > The premises of the problem remain the same (George's translated

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-08 Thread Stefano Stabellini
On Fri, 5 Jun 2015, Wei Liu wrote: > On Fri, Jun 05, 2015 at 06:10:17PM +0100, Stefano Stabellini wrote: > > On Fri, 5 Jun 2015, Wei Liu wrote: > > > Hi all > > > > > > This bug is now considered a blocker for 4.6 release. > > > > > > The premises of the problem remain the same (George's translat

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-05 Thread Wei Liu
On Fri, Jun 05, 2015 at 06:13:01PM +0100, Stefano Stabellini wrote: > On Fri, 5 Jun 2015, Ian Campbell wrote: > > On Fri, 2015-06-05 at 17:43 +0100, Wei Liu wrote: > > > > > 3. Add a libxl layer that wraps necessary information, take over > > >Andrew's work on libxl migration v2. Having a lib

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-05 Thread Ian Campbell
On Fri, 2015-06-05 at 18:10 +0100, Stefano Stabellini wrote: > On Fri, 5 Jun 2015, Wei Liu wrote: > > Hi all > > > > This bug is now considered a blocker for 4.6 release. > > > > The premises of the problem remain the same (George's translated > > version): > > > > 1. QEMU may need extra pages f

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-05 Thread Wei Liu
On Fri, Jun 05, 2015 at 06:10:17PM +0100, Stefano Stabellini wrote: > On Fri, 5 Jun 2015, Wei Liu wrote: > > Hi all > > > > This bug is now considered a blocker for 4.6 release. > > > > The premises of the problem remain the same (George's translated > > version): > > > > 1. QEMU may need extra

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-05 Thread Wei Liu
On Fri, Jun 05, 2015 at 05:58:11PM +0100, Ian Campbell wrote: > On Fri, 2015-06-05 at 17:43 +0100, Wei Liu wrote: > > > 3. Add a libxl layer that wraps necessary information, take over > >Andrew's work on libxl migration v2. Having a libxl layer that's not > >part of migration v2 is a was

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-05 Thread Andrew Cooper
On 05/06/15 17:58, Ian Campbell wrote: > On Fri, 2015-06-05 at 17:43 +0100, Wei Liu wrote: > >> 3. Add a libxl layer that wraps necessary information, take over >>Andrew's work on libxl migration v2. Having a libxl layer that's not >>part of migration v2 is a waste of effort. >> >> There a

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-05 Thread Stefano Stabellini
On Fri, 5 Jun 2015, Ian Campbell wrote: > On Fri, 2015-06-05 at 17:43 +0100, Wei Liu wrote: > > > 3. Add a libxl layer that wraps necessary information, take over > >Andrew's work on libxl migration v2. Having a libxl layer that's not > >part of migration v2 is a waste of effort. > > > >

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-05 Thread Stefano Stabellini
On Fri, 5 Jun 2015, Wei Liu wrote: > Hi all > > This bug is now considered a blocker for 4.6 release. > > The premises of the problem remain the same (George's translated > version): > > 1. QEMU may need extra pages from Xen to implement option ROMS, and so at >the moment it calls set_max_me

Re: [Xen-devel] QEMU bumping memory bug analysis

2015-06-05 Thread Ian Campbell
On Fri, 2015-06-05 at 17:43 +0100, Wei Liu wrote: > 3. Add a libxl layer that wraps necessary information, take over >Andrew's work on libxl migration v2. Having a libxl layer that's not >part of migration v2 is a waste of effort. > > There are several obstacles for libxl migration v2 at